diff --git a/docs/en/solutions/acp/How_to_Deliver_ACP_on_Bare_Metal_with_Immutable_Infrastructure.md b/docs/en/solutions/acp/How_to_Deliver_ACP_on_Bare_Metal_with_Immutable_Infrastructure.md new file mode 100644 index 000000000..a11fa8a78 --- /dev/null +++ b/docs/en/solutions/acp/How_to_Deliver_ACP_on_Bare_Metal_with_Immutable_Infrastructure.md @@ -0,0 +1,3033 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 4.4.x +--- + +# How to Deliver ACP on Bare Metal with Immutable Infrastructure + +## Purpose + +This article is the Bare Metal part of the standard delivery of Alauda Container Platform (ACP) 4.4 on +Immutable Infrastructure (Alauda OS nodes). It covers both kinds of cluster that run on physical +servers: the `global` cluster — a primary and a standby for disaster recovery (DR), including the +preparation that turns `global` into the management cluster for Bare Metal workload clusters — and +the workload clusters. Every step comes with the manifest or command to run and the check that +proves it worked. + +Read [How to Deliver ACP with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md) +first. It owns the provider-neutral parts that this article links to instead of repeating: the +[reference design](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#reference-design), +the [parameter files, `render`, `load_env` and `apply_rendered`](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#prepare-the-parameter-files), +the [delivery flow](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#delivery-flow), +the [platform plugins](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#install-the-platform-plugins), +[machine configuration](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#configure-machines), +[sizing](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#size-the-clusters) and +[acceptance](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#accept-the-delivery). +Its conventions apply here too: every command block names the cluster it runs against, and manifests +are files `/-.yaml` numbered in apply order below `50`. + +Scope: + +- **ACP 4.4** with the **Alauda Container Platform Bare Metal Infrastructure Provider v1.0.0** (the + first release of the provider, compatible with ACP 4.4, `linux/amd64` only), the Kubeadm Provider + delivered with the same release, and the Bare Metal Alauda OS archive published for the same ACP 4.4 + patch release. +- **YAML only.** The Bare Metal provider has no web console workflow for creating clusters; every + cluster step in this article is `kubectl`. +- **Physical servers** (or lab virtual machines) that boot an ISO through BMC virtual media. Nodes + that run a traditional operating system are out of scope; use the + [standard installation path](https://docs.alauda.io/container_platform/4.4/install/index.html) for + those. +- **Worker pools for applications are out of scope** — they follow the customer's machines. This + article creates the control planes and the platform `infra` pools, and shows one generic + [worker pool](#worker-pools) to start from; add the customer's pools before handover. + +Validation status: the manifests in this article are assembled from the ACP 4.4 product documentation +and the provider version named above. Validate them in a staging environment before a production +delivery. + +## Resolution + +### Bare-Metal-specific design + +#### Hosts are registered and reprovisioned, not created + +Bare Metal has no IaaS API that creates a machine. A host is installed once from an ISO, registers +itself, and from then on Cluster API borrows it for one Kubernetes node at a time +([Bare Metal Provider](https://docs.alauda.io/immutable-infra/1.0/overview/providers/bare-metal.html#concepts-and-terminology)): + +| Object | Written by | Role | +| --- | --- | --- | +| `MachineRegistration` | You, once per cluster | Registration endpoint and first-install settings, baked into the ISO | +| `SeedImage` | You, once per cluster | Builds that ISO from the `baremetal-base-image-iso` image | +| `MachineInventory` | `elemental-operator`, when a host registers | The long-lived identity of one physical host. It survives every reprovision, node deletion and cluster deletion | +| `MachineInventoryPool` | You, one per node pool | The exact inventories a pool may use — nothing outside the list is ever provisioned | +| `BaremetalMachineTemplate` | You, one per node pool | Points a `KubeadmControlPlane` or `MachineDeployment` at its pool | +| `BaremetalMachine` | The provider | Binds one `Machine` to one inventory and drives the host with `reprovision` and `clean` plans | +| `elemental-image-catalog` ConfigMap | You patch it | Maps each Kubernetes version to the `baremetal-base-image` that reprovisions a host. The provider chart creates it **empty** | + +What this means for the delivery: + +- **Manifests are applied in two passes.** Inventory names are generated at registration + (`-`), so the pools can only be written after the hosts have booted the ISO: + registration first, everything else once the inventories exist. +- **`maxSurge: 0` on every pool.** There is no spare machine to create; a replacement is + delete-then-add from the same pool. +- **A catalog miss is terminal.** A `BaremetalMachine` that hits `ImageCatalogMiss` stays `Failed` + even after you add the entry; only deleting its `Machine` recovers it. Fill the catalog before the + first cluster manifest, not after the first failure. +- **Nothing deletes a host.** Removing a node or a whole cluster returns the inventory to its pool. + +#### Control-plane endpoint: `Internal` Self-built VIP (Alive) + +Main line: `BaremetalCluster.spec.controlPlaneLoadBalancer.type: Internal`. The provider runs Alive +(keepalived, IPVS and Lease arbitration) on the control-plane nodes and moves the VIP itself, so no +load balancer is needed and the backend list follows node replacement without anyone editing it. Its +requirements fail silently when missed, so confirm each one with the network team +([Network Connectivity](https://docs.alauda.io/immutable-infra/1.0/create-cluster/bare-metal.html#network-connectivity)): + +- The VIP is an unused IPv4 address **in the same Layer-2 broadcast domain** as the control-plane + node IPs — VRRP and gratuitous ARP do not cross a router. +- The `vrid` is **unique in that Layer-2 domain**, across every cluster on the segment; a collision + leaves the VIP unreachable. +- Switches and security policy allow VRRP and gratuitous ARP, and the node image provides IPVS and + lets Alive set `net.ipv4.conf.all.arp_accept=1` and `net.ipv4.vs.conntrack=1` — checked on every + control-plane node in [Workload cluster checks](#workload-cluster-checks). +- Three control-plane nodes. The Bare Metal provider does not support a single-node control plane, + and the same three nodes arbitrate the VIP. + +Alternative: `type: External` when the customer insists on its own Layer-4 load balancer for the API. +The load balancer owner then maintains the backend list on every control-plane change — see +[External load balancer for the control-plane endpoint](#external-load-balancer-for-the-control-plane-endpoint). + +Either way the endpoint is an IP address, never a domain name: the API server certificate is issued by +the cluster's own CA, so a customer certificate cannot be served on it. The address is written into +certificates and kubeconfigs and does not change after `kubeadm init`. + +#### Persistent `/var/cpaas` is a managed data volume of the inventory + +Every node has a system disk for Alauda OS and **one data disk for `/var/cpaas`** (platform state; +monitoring and logging write under it — see the common +[reference design](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#reference-design)). +On Bare Metal the persistent disk is declared in `MachineInventory.spec.storage`, not in any Cluster +API template ([Manage Data Disks on Bare-Metal Hosts](https://docs.alauda.io/immutable-infra/1.0/how-to/manage-bare-metal-storage.html)): + +- **It belongs to the host, not to the node.** Prepare (format a blank disk, or adopt an existing + filesystem) runs while the inventory is unallocated; a reprovision mounts the volume before kubelet + starts; deleting the node unmounts it and keeps the filesystem and its data. +- **Declare it before the inventory joins a pool.** The declaration cannot change while the inventory + is allocated, and an inventory with a declaration is not `Available` until Prepare succeeds. +- **Select the disk by a stable ID** (`wwn:`, `nvme-eui:`, `nvme-nguid:`). `/dev/sdX` names change + between boots. +- **Keep each pool at exactly its replica count.** Data stays on the host and does not follow a + replacement `Machine`. With no spare inventory in the pool, the host released by a deleted node is + the only one the replacement can take, so it comes back on the same host and finds its own + `/var/cpaas`. Adding a successor host to a pool is a deliberate act — and that node starts with an + empty `/var/cpaas`. +- **Formatting needs an approval.** `InitializeIfBlank` is authorized per inventory with the + `storagectl` binary built from the same provider revision; never write the approval by hand. + +#### A reprovision wipes the system state + +Every node attach runs `cloud-init clean`, `elemental upgrade` and a reboot, after which the initramfs +clears `/var/lib/kubelet`, `/var/lib/containerd`, `/var/lib/etcd` and `/etc/kubernetes` +([host lifecycle plans](https://docs.alauda.io/immutable-infra/1.0/overview/providers/bare-metal.html#clean-reprovision)). +A Kubernetes upgrade replaces every node the same way. Therefore: + +- Nothing survives a replacement except the managed volumes. Do not leave delivery artifacts, scripts + or data anywhere else on a node. +- Control-plane nodes are replaced one at a time (`maxSurge: 0`), so etcd keeps quorum while each + member is rebuilt from its peers. +- The `COS_STATE` partition must hold two complete systems during an upgrade. Size it to 20 GiB in the + `SeedImage` at install time; it cannot be resized later without reinstalling the host + ([Size COS_STATE before the first install](https://docs.alauda.io/immutable-infra/1.0/create-cluster/bare-metal.html#size-cos-state)). + +#### Network identity is captured at first boot + +`elemental-register` records the Live ISO network — addresses, bonds, VLANs, default route and DNS — +as `MachineInventory.spec.observedNetwork`, and the first install and every reprovision replay it +([network identity](https://docs.alauda.io/immutable-infra/1.0/overview/providers/bare-metal.html#observed-network)). +The network you configure in the Live ISO **is** the node's permanent network: + +- Give each host its planned production address in the Live ISO — statically with `nmcli`, or with a + DHCP reservation. A random DHCP lease becomes the node's permanent address. +- Configure bonds and VLANs in the Live ISO the way the node must keep them. +- Check `spec.observedNetwork` against the plan before an inventory enters a pool. + +#### Every host has more than one disk + +With a system disk and a `/var/cpaas` disk, Elemental may install onto either one when +`install.device` is empty, and `/dev/sda` can name a different disk in the Live ISO than in the +installed system. Every host in this article therefore uses the fail-closed fixed-system-disk +workflow: the ISO targets `/dev/elemental-install-target`, which the operator links to the system +disk's WWN at the console before starting the installation +([Select a Fixed System Disk on Multi-Disk Bare-Metal Hosts](https://docs.alauda.io/immutable-infra/1.0/how-to/configure-fixed-install-disk-bare-metal.html)). + +#### What disaster recovery adds on Bare Metal + +On top of the increments every provider shares — one encryption configuration, the same etcd server +certificate SANs, the import ConfigMap on both sides — Bare Metal DR needs its own +([Bare Metal DR Model](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#bare-metal-dr-model)): + +| Increment | Why | +| --- | --- | +| Same ServiceAccount signing key on both sides | The workload `baremetal-system-agent` token issued by the primary must be accepted by the standby API server after a failover | +| Split system-agent identity: `splitAuthEnabled: true`; `sharedAuthReadOnly: false` on the active side, `true` on the standby | `global` hosts use a cluster-local identity; workload hosts use a shared identity that is synchronized from primary to standby | +| `baremetal.cluster.io/system-agent-auth-scope: global` on the bootstrap `MachineRegistration` | Marks the `global` hosts for the handoff | +| `handoffHook.directAPIServer: true` | After handoff, `global` hosts report to their local VIP on `6443`, not to the bootstrap node | +| `baremetal-dr-rules` ConfigMap on both sides **before** `etcd-sync` | Keeps each side's `global` plan Secrets out of synchronization. Without it, the first sync gives the standby hosts the primary's plans | +| Same trusted `thirdParty` platform certificate on both sides | Hosts and the standby reach the platform through the one domain | +| Same Registry address `:11443` on both sides | Synchronization then replicates the Registry keys as a no-op instead of overwriting the standby | + +In the main line all of these are already in the manifests and the cluster files below; +[Single cluster](#single-cluster) lists what to remove when there is no DR. + +### Parameters + +Use the parameter files and the helpers from +[Prepare the parameter files](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#prepare-the-parameter-files). +Bare Metal adds entries to `site.env` and to every cluster file, and one small file per host, because +the data disk of every host has its own ID. In every shell — on the bootstrap nodes as well as on the +delivery host — define the common helpers first, then run the commands in the delivery directory after +`load_env .env`. Commands marked `# on the bootstrap cluster` run on that side's bootstrap +node with `BOOTSTRAP_KUBECONFIG`, the kubeconfig that Step 2 of the product procedure copies to +`/root/.kube/config`. + +Common variables with a Bare Metal constraint: + +- `PLATFORM_HOST` is a domain name in the DR topology; only this domain switches during DR. +- `REGISTRY_DOMAIN` must be `:11443` on Bare Metal — the same value on both `global` + sides. +- `CLUSTER_NAME` is `global` in both `global` cluster files. The `global` manifests write the `global` + names literally, so a wrong value cannot break the naming rule. +- `K8S_VERSION` is the Kubernetes version of the Bare Metal OS archive with its leading `v` (for ACP + 4.4.1, `v1.35.6-acp.9` per the + [OS Support Matrix](https://docs.alauda.io/immutable-infra/1.0/overview/os-support-matrix.html)); it + becomes the `elemental-image-catalog` key. +- `CONTROL_PLANE_VIP` sits in the Layer-2 domain of that cluster's control-plane nodes. + +`site.env` — Bare Metal entries: + +| Variable | Meaning | Example | Where to get it | +| --- | --- | --- | --- | +| `LOCAL_REGISTRY_ADDRESS` | Bootstrap Registry as seen from the bootstrap node (pushing) | `127.0.0.1:11443` | Fixed product value | +| `BOOTSTRAP_REGISTRY_ADDRESS` | Bootstrap Registry as seen from inside the bootstrap cluster (AppRelease `repoURL`, bootstrap `SeedImage`) | `172.18.0.1:11443` | Fixed product value | +| `INGRESS_CLASS_NAME` | Ingress class of the platform's built-in bootstrap ingress, referenced by the provider AppReleases | `global-alb2` | Fixed product value; not an application-ingress choice | +| `KUBEADM_PROVIDER_PACK` | Kubeadm Provider package on the bootstrap node | `/root/pkgs/cluster-api-provider-kubeadm.amd64.vX.Y.Z.tgz` | Customer Portal | +| `BAREMETAL_PROVIDER_PACK` | Bare Metal Infrastructure Provider package on the bootstrap node | `/root/pkgs/cluster-api-provider-baremetal.amd64.vX.Y.Z.tgz` | Customer Portal | +| `KUBEADM_PROVIDER_VERSION` | Chart version of the Kubeadm Provider | `vX.Y.Z` | Tag list of `ait/chart-cluster-api-provider-kubeadm` after the push in [Prepare the bootstrap cluster](#prepare-the-bootstrap-cluster) | +| `BAREMETAL_PROVIDER_VERSION` | Chart version of the Bare Metal provider | `vX.Y.Z` | Tag list of `ait/chart-cluster-api-provider-baremetal` after the same push | +| `OS_IMAGE_ARCHIVE` | Bare Metal Alauda OS archive (one OCI layout holding two images) | `/root/pkgs/baremetal-os.tar` | Customer Portal OS download page, Bare Metal entry | +| `OS_IMAGE_TAG` | Tag shared by `baremetal-base-image` and `baremetal-base-image-iso` | `vX.Y.Z` | `ctr -n bare-metal-os images ls -q` after the import | +| `DNS_IMAGE_TAG` | CoreDNS image tag preloaded in the OS image | `1.14.7-v4.4.7` | `coredns` column of the OS Support Matrix row of your ACP release | +| `ETCD_IMAGE_TAG` | etcd image tag preloaded in the OS image | `v3.6.13-acp.2` | `etcd` column of the same row | +| `KUBE_OVN_VERSION` | Kube-OVN **chart** version pinned on each `Cluster` | `v4.4.12` | `kube-ovn (chart)` column of the same row — not the Kube-OVN component version | +| `PRIMARY_CLUSTER_VIP` | Control-plane VIP of the primary `global` (DR) | `192.0.2.10` | Network plan; equals `CONTROL_PLANE_VIP` of `global-primary.env` | +| `STANDBY_CLUSTER_VIP` | Control-plane VIP of the standby `global` (DR) | `203.0.113.10` | Network plan; equals `CONTROL_PLANE_VIP` of `global-standby.env` | +| `ENCRYPTION_PROVIDER_CONF` | Path of the shared encryption configuration on both bootstrap nodes (DR) | `/root/global-dr/encryption-provider.conf` | Created in [Generate the shared DR material](#generate-the-shared-dr-material) | +| `BM_STORAGECTL` | Path of the `storagectl` binary | `/root/pkgs/storagectl` | See [Download the materials](#download-the-materials) | + +Cluster files — the same names in every file: + +| Variable | Meaning | Example (`global-primary` / `global-standby` / `wl-prod`) | Where to get it | +| --- | --- | --- | --- | +| `BOOTSTRAP_KUBECONFIG` | Kubeconfig of this side's bootstrap cluster, on its bootstrap node (`global` files only) | `/root/.kube/config` / `/root/.kube/config` / — | Copied from `/var/cpaas/data/alauda.kubeconfig` in [Prepare the bootstrap cluster](#prepare-the-bootstrap-cluster) | +| `HOST_IP` | Static IP of this side's bootstrap node (`global` files only) | `192.0.2.5` / `203.0.113.5` / — | Bootstrap node | +| `NODE_REGISTRY_ADDRESS` | `:11443`, the Registry the `global` hosts pull from during installation (`global` files only) | `192.0.2.5:11443` / `203.0.113.5:11443` / — | Derived from `HOST_IP` | +| `BM_CONTROL_PLANE_VRID` | Keepalived `virtual_router_id` of this cluster's VIP | `51` / `52` / `61` | Unique in the control-plane Layer-2 domain | +| `BM_DIRECT_API_SERVER` | Hand the `global` hosts off to the local VIP (`global` files only) | `true` / `true` / — | `true` for DR, `false` for a single cluster | +| `BM_SPLIT_AUTH_ENABLED` | Split `global` and workload system-agent identities (`global` files only) | `true` / `true` / — | `true` for DR, `false` for a single cluster | +| `BM_SHARED_AUTH_READ_ONLY` | Shared workload identity is read-only on this side (`global` files only) | `false` / `true` / — | `false` on the active side, `true` on the standby; `false` for a single cluster | +| `BM_CP_INVENTORY_1` | `MachineInventory` of control-plane host 1 | `global-00000000-0000-0000-0000-000000000001` | Registration, see [Record the inventories](#record-the-inventories) | +| `BM_CP_INVENTORY_2` | `MachineInventory` of control-plane host 2 | `global-00000000-0000-0000-0000-000000000002` | Registration | +| `BM_CP_INVENTORY_3` | `MachineInventory` of control-plane host 3 | `global-00000000-0000-0000-0000-000000000003` | Registration | +| `BM_CP_HOSTNAME_1` | Node hostname of control-plane host 1 | `primary-global-cp-1` / `standby-global-cp-1` / `wl-prod-cp-1` | Naming plan; the side prefix tells the DR pair apart at a glance | +| `BM_CP_HOSTNAME_2` | Node hostname of control-plane host 2 | `primary-global-cp-2` | Naming plan | +| `BM_CP_HOSTNAME_3` | Node hostname of control-plane host 3 | `primary-global-cp-3` | Naming plan | +| `BM_INFRA_INVENTORY_1` | `MachineInventory` of `infra` host 1 | `global-00000000-0000-0000-0000-000000000004` | Registration | +| `BM_INFRA_INVENTORY_2` | `MachineInventory` of `infra` host 2 | `global-00000000-0000-0000-0000-000000000005` | Registration | +| `BM_INFRA_INVENTORY_3` | `MachineInventory` of `infra` host 3 | `global-00000000-0000-0000-0000-000000000006` | Registration | +| `BM_INFRA_HOSTNAME_1` | Node hostname of `infra` host 1 | `primary-global-infra-1` | Naming plan | +| `BM_INFRA_HOSTNAME_2` | Node hostname of `infra` host 2 | `primary-global-infra-2` | Naming plan | +| `BM_INFRA_HOSTNAME_3` | Node hostname of `infra` host 3 | `primary-global-infra-3` | Naming plan | +| `BM_INGRESS_INVENTORY_1` | [Dedicated pools](#dedicated-pools) only: `MachineInventory` of `ingress` host 1; `_2`, `_3` and the `MONITOR` / `LOG` equivalents follow the same pattern | — / — / `wl-prod-00000000-0000-0000-0000-000000000021` | Registration | +| `BM_WORKER_INVENTORY_1` | [Worker pools](#worker-pools) only: `MachineInventory` of worker host 1; `_2` and `_3` follow the same pattern | — / — / `wl-prod-00000000-0000-0000-0000-000000000031` | Registration | +| `BM_WORKER_HOSTNAME_1` | [Worker pools](#worker-pools) only: node hostname of worker host 1, same pattern | — / — / `wl-prod-worker-1` | Naming plan | +| `BM_INGRESS_HOSTNAME_1` | [Dedicated pools](#dedicated-pools) only: node hostname of `ingress` host 1, same pattern | — / — / `wl-prod-ingress-1` | Naming plan | + +Host files — one per host, `hosts//-.env`: + +| Variable | Meaning | Example | Where to get it | +| --- | --- | --- | --- | +| `BM_INVENTORY` | `MachineInventory` of this host | `global-00000000-0000-0000-0000-000000000001` | Registration | +| `BM_DATA_DISK_ID` | Stable ID of the `/var/cpaas` data disk | `wwn:0x5000000000000001` | `status.observedStorage` of the inventory, see [Prepare /var/cpaas on every inventory](#prepare-varcpaas-on-every-inventory) | +| `BM_DATA_DISK_MIN_SIZE` | Smallest size the disk must have; Prepare refuses a smaller device | `200Gi` | [Size the clusters](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#size-the-clusters) | + +Session variables — set by the commands of this article, never stored in a file: + +| Variable | Meaning | Example | Where to get it | +| --- | --- | --- | --- | +| `INSTALLER_IP` | Pod IP of the installer in the bootstrap cluster | (set at run time) | `kubectl get pods -n cpaas-system -l service_name=cpaas-installer` | +| `ENCRYPTION_PROVIDER_SECRET_B64` | The shared etcd encryption key (DR) | (never stored) | Generated once in [Generate the shared DR material](#generate-the-shared-dr-material); never written to a file other than `ENCRYPTION_PROVIDER_CONF` | +| `GLOBAL_REGISTRY_ADDRESS` | Registry address recorded in `ProductBase` | `acp.example.com:11443` | Read from `ProductBase` | +| `GLOBAL_REGISTRY_PASSWORD` | `admin` password of that Registry | (never stored) | Read from `cpaas-system/registry-admin`; held in the shell only | +| `BM_NS` | Namespace of every Bare Metal object | `cpaas-system` | Fixed | +| `BM_STORAGE_FILE` | Rendered storage declaration of one host | `/tmp/baremetal-storage.yaml` | Working file | +| `BM_INVENTORY_FILE` | Live copy of one inventory | `/tmp/baremetal-machineinventory.json` | Working file | +| `BM_PATCH_FILE` | Patch generated by `storagectl` | `/tmp/baremetal-storage.patch.json` | Working file | +| `PRIMARY_GLOBAL_KUBECONFIG` | Kubeconfig of the primary `global` (DR checks) | `/root/.kube/acp/global-primary.kubeconfig` | `GLOBAL_KUBECONFIG` of `global-primary.env` | +| `STANDBY_GLOBAL_KUBECONFIG` | Kubeconfig of the standby `global` (DR checks) | `/root/.kube/acp/global-standby.kubeconfig` | `GLOBAL_KUBECONFIG` of `global-standby.env` | +| `PRIMARY_BOOTSTRAP_KUBECONFIG` | Copy, on the delivery host, of the primary `BOOTSTRAP_KUBECONFIG` (DR checks) | `/root/.kube/acp/bootstrap-primary.kubeconfig` | `/root/.kube/config` of the primary bootstrap node | +| `STANDBY_BOOTSTRAP_KUBECONFIG` | Copy, on the delivery host, of the standby `BOOTSTRAP_KUBECONFIG` (DR checks) | `/root/.kube/acp/bootstrap-standby.kubeconfig` | The same file on the standby bootstrap node | +| `PRIMARY_GLOBAL_HOSTS` | Management addresses of the six primary `global` hosts | `"192.0.2.21 ... 192.0.2.26"` | Network plan | +| `STANDBY_GLOBAL_HOSTS` | Management addresses of the six standby `global` hosts | `"203.0.113.21 ... 203.0.113.26"` | Network plan | +| `PRIMARY_GLOBAL_REGISTRATION_NAME` | Bootstrap `MachineRegistration` of the primary | `global-registration` | `10-registration.yaml` | +| `STANDBY_GLOBAL_REGISTRATION_NAME` | Bootstrap `MachineRegistration` of the standby | `global-registration` | `10-registration.yaml` | + +Example entries. The addresses are documentation addresses; replace every value. + +```bash +# site.env — Bare Metal entries, appended to the common entries +LOCAL_REGISTRY_ADDRESS='127.0.0.1:11443' +BOOTSTRAP_REGISTRY_ADDRESS='172.18.0.1:11443' +INGRESS_CLASS_NAME='global-alb2' +KUBEADM_PROVIDER_PACK='/root/pkgs/cluster-api-provider-kubeadm.amd64.vX.Y.Z.tgz' +BAREMETAL_PROVIDER_PACK='/root/pkgs/cluster-api-provider-baremetal.amd64.vX.Y.Z.tgz' +KUBEADM_PROVIDER_VERSION='vX.Y.Z' +BAREMETAL_PROVIDER_VERSION='vX.Y.Z' +OS_IMAGE_ARCHIVE='/root/pkgs/baremetal-os.tar' +OS_IMAGE_TAG='vX.Y.Z' +DNS_IMAGE_TAG='1.14.7-v4.4.7' +ETCD_IMAGE_TAG='v3.6.13-acp.2' +KUBE_OVN_VERSION='v4.4.12' +PRIMARY_CLUSTER_VIP='192.0.2.10' +STANDBY_CLUSTER_VIP='203.0.113.10' +ENCRYPTION_PROVIDER_CONF='/root/global-dr/encryption-provider.conf' +BM_STORAGECTL='/root/pkgs/storagectl' +``` + +```bash +# global-primary.env — Bare Metal entries, appended to the common entries +# (CLUSTER_NAME='global', CONTROL_PLANE_VIP='192.0.2.10', the CIDRs and both kubeconfigs are common entries) +BOOTSTRAP_KUBECONFIG='/root/.kube/config' +HOST_IP='192.0.2.5' +NODE_REGISTRY_ADDRESS='192.0.2.5:11443' +BM_CONTROL_PLANE_VRID='51' +BM_DIRECT_API_SERVER='true' +BM_SPLIT_AUTH_ENABLED='true' +BM_SHARED_AUTH_READ_ONLY='false' +# filled in after registration +BM_CP_INVENTORY_1='global-00000000-0000-0000-0000-000000000001' +BM_CP_INVENTORY_2='global-00000000-0000-0000-0000-000000000002' +BM_CP_INVENTORY_3='global-00000000-0000-0000-0000-000000000003' +BM_CP_HOSTNAME_1='primary-global-cp-1' +BM_CP_HOSTNAME_2='primary-global-cp-2' +BM_CP_HOSTNAME_3='primary-global-cp-3' +BM_INFRA_INVENTORY_1='global-00000000-0000-0000-0000-000000000004' +BM_INFRA_INVENTORY_2='global-00000000-0000-0000-0000-000000000005' +BM_INFRA_INVENTORY_3='global-00000000-0000-0000-0000-000000000006' +BM_INFRA_HOSTNAME_1='primary-global-infra-1' +BM_INFRA_HOSTNAME_2='primary-global-infra-2' +BM_INFRA_HOSTNAME_3='primary-global-infra-3' +``` + +`global-standby.env` has the same names with the standby values: `HOST_IP='203.0.113.5'`, +`NODE_REGISTRY_ADDRESS='203.0.113.5:11443'`, its own `vrid`, inventories and hostnames, and +`BM_SHARED_AUTH_READ_ONLY='true'`. + +```bash +# wl-prod.env — Bare Metal entries, appended to the common entries; no bootstrap or DR entries +BM_CONTROL_PLANE_VRID='61' +BM_CP_INVENTORY_1='wl-prod-00000000-0000-0000-0000-000000000011' +BM_CP_INVENTORY_2='wl-prod-00000000-0000-0000-0000-000000000012' +BM_CP_INVENTORY_3='wl-prod-00000000-0000-0000-0000-000000000013' +BM_CP_HOSTNAME_1='wl-prod-cp-1' +BM_CP_HOSTNAME_2='wl-prod-cp-2' +BM_CP_HOSTNAME_3='wl-prod-cp-3' +BM_INFRA_INVENTORY_1='wl-prod-00000000-0000-0000-0000-000000000014' +BM_INFRA_INVENTORY_2='wl-prod-00000000-0000-0000-0000-000000000015' +BM_INFRA_INVENTORY_3='wl-prod-00000000-0000-0000-0000-000000000016' +BM_INFRA_HOSTNAME_1='wl-prod-infra-1' +BM_INFRA_HOSTNAME_2='wl-prod-infra-2' +BM_INFRA_HOSTNAME_3='wl-prod-infra-3' +``` + +```bash +# hosts/global-primary/cp-1.env — one file per host +BM_INVENTORY='global-00000000-0000-0000-0000-000000000001' +BM_DATA_DISK_ID='wwn:0x5000000000000001' +BM_DATA_DISK_MIN_SIZE='200Gi' +``` + +Intentional `${...}` literals: `${System Information/UUID}` and `${System Information/Serial Number}` +in the two `10-registration.yaml` files are expanded by `elemental-register` from SMBIOS data on each +host. `render` leaves them alone because they are not parameter names, so those two files are applied +with a check that filters them out instead of with `apply_rendered`. + +### Prepare Bare Metal + +#### Collect the facts + +| Scope | Fact | Why it matters | +| --- | --- | --- | +| Per host | Serial number, SMBIOS UUID, BMC address | Maps a registered inventory to its role. BMC credentials stay with the customer and never enter a file | +| Per host | System disk WWN, data disk stable ID and size, any disk that must keep data | Install target, `/var/cpaas` declaration, and what must never be wiped | +| Per host | NICs, bonds, VLANs, planned IP, gateway, DNS | Becomes the permanent network at first boot | +| Per host | Real hardware TPM present (`/dev/tpm0`) or not | Decides `emulate-tpm` | +| Per cluster | Control-plane VIP and `vrid`, Pod / Service / Join CIDRs | Must not overlap each other, the host network or another cluster | +| Site | Platform domain and its DNS record, internal DNS and NTP servers, platform certificate (DR: one trusted certificate for the domain) | Registration, the clock gate, DR | + +#### Prepare every host + +Do this on every host **before** it boots the ISO; each item is a failure that otherwise surfaces much +later ([Host Disk and Boot Preparation](https://docs.alauda.io/immutable-infra/1.0/create-cluster/bare-metal.html#host-disk-and-boot-preparation)): + +- **CPU baseline.** The Alauda OS images require `x86-64-v2` on x86_64 hosts. Check the CPU model + against the vendor specification, or run this in the Live ISO console — no output means the + baseline is met: + +```bash +# in the Live ISO console +for f in cx16 lahf_lm popcnt pni sse4_1 sse4_2 ssse3; do grep -qw "$f" /proc/cpuinfo || echo "missing $f"; done +``` + +- **Disks.** In the RAID controller, present one system disk and one `/var/cpaas` data disk per host. + Classify every disk or virtual disk; wipe the system disk and any obsolete boot disk so that no old + operating system can boot; remove residual `COS_STATE`, `COS_PERSISTENT`, `COS_OEM` and + `COS_RECOVERY` labels from **all** disks and multipath paths — a stale label on a second disk makes + Elemental resolve the wrong device. Never wipe a disk that holds data to keep. +- **Boot.** Enable BMC virtual media and boot the virtual CD for the installation. After the + installation, eject the media and restore disk-first boot. +- **TPM.** A host with a working hardware TPM can register with `emulate-tpm: false`; every host + without one — virtual machines and TPM-less servers alike — needs `emulate-tpm: true` with + `emulated-tpm-seed: -1`. The manifests below use emulation; switch a cluster's registration to + `false` only when every host of that cluster has a hardware TPM. A host without a TPM and without + emulation never sends its registration request. + +#### Pass the clock gate + +Every host must carry the correct time **before** it boots the ISO +([Host Time Synchronization](https://docs.alauda.io/immutable-infra/1.0/create-cluster/bare-metal.html#host-time-synchronization)). +The whole cluster PKI inherits the clock of the host that runs `kubeadm init`: a host running ahead +issues certificates that every other node treats as not yet valid, and the control plane never comes +up. Nothing on this path checks it for you. + +1. Make the site NTP servers reachable from the host subnet (UDP `123`). Do not depend on public NTP. +2. Set the clock in the BIOS/UEFI setup or through the BMC, with the hardware clock on UTC — the disk + has no operating system yet to run a command in. +3. Compare the hosts with each other, not only with a reference: the spread across all hosts of one + cluster must stay within 10 seconds before any of them boots the ISO. +4. After the nodes are `Ready`, confirm with `timedatectl` and `chronyc tracking` on each node. The + NTP `MachineConfig` of [Configure machines](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#configure-machines) + keeps the time right afterwards; it cannot fix a host that is still installing. + +#### Open the network flows by stage + +Bare Metal hosts are installed and joined entirely over the network, so every path must be open +before the first host boots. Approve the flows from the authoritative tables, not from a copy: + +| Stage | Flows | Authority | +| --- | --- | --- | +| `global` installation (bootstrap phase) | `global` hosts → bootstrap node `12443` (registration) and `11443` (images); bootstrap node → `global` VIP `6443` | [Bootstrap node network](https://docs.alauda.io/immutable-infra/1.0/global/install.html#bootstrap-host-network) | +| Workload hosts: first boot, provisioning, join, steady state | DNS, NTP, platform HTTPS (registration and plan polling), Registry `11443`, the cluster's own endpoint, node-to-node traffic | [Network Connectivity](https://docs.alauda.io/immutable-infra/1.0/create-cluster/bare-metal.html#network-connectivity) | +| `global` hosts after handoff (DR) | Each `global` host → its local VIP `6443` | [Endpoint and Identity](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#endpoint-and-identity) | +| `global` ↔ workload management | `global` nodes → every workload endpoint `6443`; in DR from both `global` sides | [Connecting global Cluster and Workload Cluster](https://docs.alauda.io/container_platform/4.4/configure/clusters/on-premises.html#connecting-global-cluster-and-workload-cluster) | +| Between the two `global` sides | `443`, `6443`, `11443`, `2379` (and `80` for HTTP access), in **both** directions | [DR Network Requirements](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#network-requirements) | +| Inside a cluster | Component port inventory | [ACP 4.4 Communication Matrix](https://docs.alauda.io/container_platform/4.4/install/prepare/communication_matrix.html) | +| Self-built VIP | VRRP and gratuitous ARP inside the control-plane Layer-2 domain | [Plan the Control Plane Endpoint](https://docs.alauda.io/immutable-infra/1.0/how-to/control-plane-endpoint.html) | + +An approval that covers only `443` produces hosts that register and then fail during install or join, +where the cause is much harder to see. The Alauda OS host firewall is maintained by the platform; the +flows above are for site firewalls, security groups and router ACLs. + +#### Provide the bootstrap nodes + +Two bootstrap nodes for the DR topology — one per side, never one reused for both — and one for a +single cluster ([Bootstrap Node Requirements](https://docs.alauda.io/immutable-infra/1.0/global/install.html#bootstrap-host-requirements)): +4 cores, 8 GB and 300 GB or more on x86_64, a traditional 64-bit Linux with Bash, `root`, `curl`, +`jq` and `openssl`. Give each a **static IP** for the whole installation: `HOST_IP` is written into +the registration endpoint, its certificate SAN and the Registry annotation the hosts pull from, and a +change part-way through breaks node provisioning. + +#### Download the materials + +Besides the Core Package and the plugin packages of the common article (which include `etcd-sync` +for DR): the Kubeadm Provider package, the Bare Metal Infrastructure Provider package, the Bare Metal +OS archive of the same ACP release +([Downloading Alauda OS](https://docs.alauda.io/immutable-infra/1.0/overview/os-support-matrix.html#downloading-alauda-os)), +and the `storagectl` binary built from exactly the same revision as the Bare Metal provider package, +because it reuses the provider's validation and approval-hash code. The product documentation does not +name a download location for `storagectl`; request the matching binary from Alauda technical support +together with the provider version, and never use one from another release. Put the two provider +packages into `packages/plugins/` so that the common upload step pushes them to `global` as well. Take +both OS images from the same archive; never pair images from different archives. + +#### Generate the shared DR material + +DR only. Generate the encryption configuration and the ServiceAccount signing key **once**, on the +primary bootstrap node, before either installation. Both sides must use the same files: a different +encryption key makes synchronized Secrets undecryptable on the standby, and a different signing key +makes every workload host fail authentication after a failover — both show only at the first failover +drill. + +```bash +# on the primary bootstrap node +load_env global-primary.env +umask 077 +mkdir -p /root/global-dr +ENCRYPTION_PROVIDER_SECRET_B64="$(head -c 32 /dev/urandom | base64)" +cat > "${ENCRYPTION_PROVIDER_CONF}" <:/root/`) and compare the three digests there. +These files are credentials: keep an access-controlled offline copy, keep them out of the delivery +directory and the archive, and never print their content. + +#### Prepare the bootstrap cluster + +Do this on **each** bootstrap node with its own cluster file — Steps 2 and 3 of +[Installing the global Cluster](https://docs.alauda.io/immutable-infra/1.0/global/install.html). + +Create the bootstrap cluster: + +```bash +# on the bootstrap node +mkdir -p /root/cpaas-install +tar -xvf -C /root/cpaas-install +cd /root/cpaas-install/installer +bash setup.sh +mkdir -p "$HOME/.local/bin" ~/.kube + +if ! command -v kubectl >/dev/null 2>&1; then + nerdctl cp minialauda-control-plane:/usr/bin/kubectl \ + "$HOME/.local/bin/kubectl" + chmod 0755 "$HOME/.local/bin/kubectl" + export PATH="$HOME/.local/bin:$PATH" +fi + +cp /var/cpaas/data/alauda.kubeconfig ~/.kube/config # = BOOTSTRAP_KUBECONFIG of this side +kubectl get nodes +``` + +The bootstrap Registry is anonymous by default. If you configured Registry credentials during +`setup.sh`, add `spec.source.chartPullSecret: global-registry-auth` and +`spec.values.global.registry.imagePullSecrets: [global-registry-auth]` to both AppReleases below and +`-u ` to the `ctr` pushes (`ctr` then prompts for the password); otherwise add nothing. + +Push the provider packages and read their chart versions into `site.env`: + +```bash +# on the bootstrap node, in the delivery directory +load_env global-primary.env +/root/cpaas-install/installer/res/amd64/packtool pack push \ + -r "${LOCAL_REGISTRY_ADDRESS}" -c "${BAREMETAL_PROVIDER_PACK}" +/root/cpaas-install/installer/res/amd64/packtool pack push \ + -r "${LOCAL_REGISTRY_ADDRESS}" -c "${KUBEADM_PROVIDER_PACK}" + +curl -s "http://${LOCAL_REGISTRY_ADDRESS}/v2/ait/chart-cluster-api-provider-kubeadm/tags/list" +curl -s "http://${LOCAL_REGISTRY_ADDRESS}/v2/ait/chart-cluster-api-provider-baremetal/tags/list" +``` + +Import both OS images and push them to the bootstrap Registry. `/var/lib/containerd` needs free space +of at least the archive size. Keep the images in the `bare-metal-os` namespace: they are pushed again to +the `global` Registry after installation. + +```bash +# on the bootstrap node +ctr -n bare-metal-os images import --all-platforms --no-unpack "${OS_IMAGE_ARCHIVE}" +ctr -n bare-metal-os images ls -q # both images, one tag: set OS_IMAGE_TAG in site.env, then load_env again + +for repo in baremetal-base-image baremetal-base-image-iso; do + ctr -n bare-metal-os images push --plain-http \ + "${LOCAL_REGISTRY_ADDRESS}/tkestack/${repo}:${OS_IMAGE_TAG}" "${repo}:${OS_IMAGE_TAG}" + curl -s "http://${LOCAL_REGISTRY_ADDRESS}/v2/tkestack/${repo}/tags/list" +done +``` + +The bootstrap registration endpoint `https://:12443` needs a serving certificate with +`HOST_IP` as an IP SAN before the provider is installed; `elemental-system-agent` verifies it in +`strict` mode. This certificate belongs to the bootstrap phase only — it is never imported into the +final cluster. + +`global-primary/01-bootstrap-dex-tls.yaml`: + +```yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: baremetal-bootstrap-selfsigned + namespace: cpaas-system +spec: + selfSigned: {} +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: baremetal-bootstrap-ca + namespace: cpaas-system +spec: + secretName: baremetal-bootstrap-ca + commonName: baremetal-bootstrap-ca + duration: 87600h + renewBefore: 720h + isCA: true + privateKey: + algorithm: RSA + size: 2048 + usages: + - cert sign + - crl sign + issuerRef: + name: baremetal-bootstrap-selfsigned + kind: Issuer +--- +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: baremetal-bootstrap-ca + namespace: cpaas-system +spec: + ca: + secretName: baremetal-bootstrap-ca +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: dex-tls-bootstrap + namespace: cpaas-system +spec: + secretName: dex.tls + commonName: ${HOST_IP} + duration: 87600h + renewBefore: 720h + privateKey: + algorithm: RSA + size: 2048 + usages: + - digital signature + - key encipherment + - server auth + ipAddresses: + - "${HOST_IP}" + - "127.0.0.1" + dnsNames: + - global-alb2 + - global-alb2.cpaas-system.svc + - global-alb2.cpaas-system.svc.cluster.local + issuerRef: + name: baremetal-bootstrap-ca + kind: Issuer +``` + +```bash +# on the bootstrap cluster +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" get crd certificates.cert-manager.io issuers.cert-manager.io +apply_rendered global-primary.env global-primary/01-bootstrap-dex-tls.yaml "${BOOTSTRAP_KUBECONFIG}" +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system wait certificate/baremetal-bootstrap-ca --for=condition=Ready --timeout=120s +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system wait certificate/dex-tls-bootstrap --for=condition=Ready --timeout=120s + +# expect ca.crt,tls.crt,tls.key — list the keys, never print the values +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get secret dex.tls -o json | jq -r '.data | keys | join(",")' +tmp_dir=$(mktemp -d) +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get secret dex.tls -o jsonpath='{.data.ca\.crt}' | base64 -d > "${tmp_dir}/ca.crt" +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get secret dex.tls -o jsonpath='{.data.tls\.crt}' | base64 -d > "${tmp_dir}/tls.crt" +openssl verify -CAfile "${tmp_dir}/ca.crt" "${tmp_dir}/tls.crt" +openssl x509 -in "${tmp_dir}/tls.crt" -noout -text | grep "IP Address:${HOST_IP}" +echo | openssl s_client -connect "${HOST_IP}:12443" -servername "${HOST_IP}" \ + -CAfile "${tmp_dir}/ca.crt" -verify_return_error 2>&1 | grep "Verify return code: 0 (ok)" +rm -rf "${tmp_dir}" +``` + +If the last check still sees an older certificate, run +`kubectl -n cpaas-system rollout restart deploy/global-alb2` and check again. + +DR only — create the two shared Secrets in this bootstrap cluster from the shared files, with +`kubectl create` as the common [Credentials](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#credentials) +rule requires. + +```bash +# on the bootstrap cluster (both sides) +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system create secret generic global-encryption-provider-config \ + --from-file=encryption-provider.conf="${ENCRYPTION_PROVIDER_CONF}" +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system create secret generic global-sa-signing-key \ + --from-file=sa.key=/root/global-dr/sa.key \ + --from-file=sa.pub=/root/global-dr/sa.pub +``` + +Install the two providers. The DR values come from the cluster file, so each side gets its +role-specific values without a later patch, and the catalog entry for `K8S_VERSION` is filled at the +same time — without it every `global` `BaremetalMachine` fails with `ImageCatalogMiss`. + +`global-primary/02-provider-appreleases.yaml`: + +```yaml +apiVersion: operator.alauda.io/v1alpha1 +kind: AppRelease +metadata: + annotations: + auto-recycle: "true" + interval-sync: "true" + name: cluster-api-provider-kubeadm + namespace: cpaas-system +spec: + destination: + cluster: "" + namespace: "" + source: + charts: + - name: ait/chart-cluster-api-provider-kubeadm + releaseName: cluster-api-provider-kubeadm + targetRevision: ${KUBEADM_PROVIDER_VERSION} + repoURL: ${BOOTSTRAP_REGISTRY_ADDRESS} + timeout: 120 + values: + global: + albName: ${INGRESS_CLASS_NAME} + auth: + default_admin: admin@cpaas.io + cluster: + isGlobal: true + name: global + networkType: kube-ovn + type: Baremetal + host: ${PLATFORM_HOST} + ingress: + ingressClassName: ${INGRESS_CLASS_NAME} + labelBaseDomain: cpaas.io + namespace: cpaas-system + platformUrl: https://${PLATFORM_HOST} + protectSecretFiles: + enabled: false + region: global + registry: + address: ${BOOTSTRAP_REGISTRY_ADDRESS} + replicas: 1 + scheme: https +--- +apiVersion: operator.alauda.io/v1alpha1 +kind: AppRelease +metadata: + annotations: + auto-recycle: "true" + interval-sync: "true" + name: cluster-api-provider-baremetal + namespace: cpaas-system +spec: + destination: + cluster: "" + namespace: "" + source: + charts: + - name: ait/chart-cluster-api-provider-baremetal + releaseName: cluster-api-provider-baremetal + targetRevision: ${BAREMETAL_PROVIDER_VERSION} + repoURL: ${BOOTSTRAP_REGISTRY_ADDRESS} + timeout: 120 + values: + global: + albName: ${INGRESS_CLASS_NAME} + auth: + default_admin: admin@cpaas.io + cluster: + isGlobal: true + name: global + networkType: kube-ovn + type: Baremetal + host: ${PLATFORM_HOST} + ingress: + ingressClassName: ${INGRESS_CLASS_NAME} + tls: + secretName: dex.tls + labelBaseDomain: cpaas.io + namespace: cpaas-system + # Bootstrap phase: registration goes to this bootstrap node, not to the platform domain. + platformUrl: https://${HOST_IP} + protectSecretFiles: + enabled: false + region: global + registry: + address: ${BOOTSTRAP_REGISTRY_ADDRESS} + replicas: 1 + scheme: https + provider: + imageCatalog: + images: + ${K8S_VERSION}: + repository: tkestack/baremetal-base-image + tag: ${OS_IMAGE_TAG} + handoffHook: + directAPIServer: ${BM_DIRECT_API_SERVER} + controlPlaneVIP: ${CONTROL_PLANE_VIP} + delivery: + enabled: true + mode: always + elemental: + server: + url: https://${HOST_IP}:12443 + systemAgent: + authMode: shared + splitAuthEnabled: ${BM_SPLIT_AUTH_ENABLED} + serviceAccountName: baremetal-system-agent + globalServiceAccountName: baremetal-global-system-agent + sharedAuthReadOnly: ${BM_SHARED_AUTH_READ_ONLY} + tls: + agentTLSMode: strict + caCertSecretName: dex.tls + caCertSecretKey: ca.crt +``` + +```bash +# on the bootstrap cluster +apply_rendered global-primary.env global-primary/02-provider-appreleases.yaml "${BOOTSTRAP_KUBECONFIG}" + +for crd in kubeadmcontrolplanes.controlplane.cluster.x-k8s.io \ + baremetalclusters.infrastructure.cluster.x-k8s.io \ + machineinventories.elemental.cattle.io; do + until kubectl get crd "${crd}" >/dev/null 2>&1; do sleep 10; done +done + +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get pods | grep -E 'cluster-api-provider-baremetal|elemental' +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system logs deploy/elemental-operator --tail=100 +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get configmap elemental-image-catalog -o jsonpath='{.data}'; echo # contains K8S_VERSION + +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get apprelease cluster-api-provider-baremetal -o json | \ + jq -e --argjson direct "${BM_DIRECT_API_SERVER}" --argjson split "${BM_SPLIT_AUTH_ENABLED}" \ + --argjson ro "${BM_SHARED_AUTH_READ_ONLY}" ' + .spec.values.handoffHook.directAPIServer == $direct and + .spec.values.elemental.systemAgent.splitAuthEnabled == $split and + .spec.values.elemental.systemAgent.sharedAuthReadOnly == $ro' +``` + +Continue only when the catalog lists `K8S_VERSION` and the last command prints `true`. A +`CrashLoopBackOff` with an unknown flag such as `--system-agent-auth-mode` means the chart and the +`elemental-operator` image come from different release payloads. + +Enable `Machine`-to-`Node` label synchronization on the bootstrap cluster's Cluster API controller. +It provisions the `global` nodes, and the `infra` role label reaches the nodes only through it; without +it every plugin pinned to `infra` stays `Pending`, which looks exactly like a wrong label +([Configure Annotations, Labels, and Taints on Nodes](https://docs.alauda.io/immutable-infra/1.0/how-to/node-configuration.html)). + +```bash +# on the bootstrap cluster +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get deploy capi-controller-manager \ + -o jsonpath='{.spec.template.spec.containers[0].args}'; echo +# only if no --additional-sync-machine-labels argument is present yet (otherwise extend its value): +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system patch deploy capi-controller-manager --type json \ + -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--additional-sync-machine-labels=node-role.kubernetes.io/.*"}]' +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system rollout status deploy/capi-controller-manager +``` + +### Install the global cluster + +This section installs one `global` side: three control-plane hosts and three `infra` hosts. In the DR +main line it runs for the primary with `global-primary.env` while the platform domain resolves to +`PRIMARY_CLUSTER_VIP`, then for the standby as described in +[Add disaster recovery](#add-disaster-recovery). The standby uses an unchanged copy of the +`global-primary/` directory — every difference between the sides is in the cluster file. + +| File | Applied to | Content | +| --- | --- | --- | +| `01-bootstrap-dex-tls.yaml`, `02-provider-appreleases.yaml` | Bootstrap cluster | [Prepare the bootstrap cluster](#prepare-the-bootstrap-cluster) | +| `10-registration.yaml` | Bootstrap cluster | `MachineRegistration` and `SeedImage` | +| `15-var-cpaas-storage.yaml` | Input of `storagectl`, once per host | `/var/cpaas` declaration | +| `20-pools.yaml` | Bootstrap cluster | Two `MachineInventoryPool` objects | +| `30-control-plane.yaml` | Bootstrap cluster | `BaremetalCluster`, control-plane template, `KubeadmControlPlane`, `Cluster` | +| `40-infra-pool.yaml` | Bootstrap cluster | `infra` template, bootstrap template, `MachineDeployment` | +| `45-import-extra-resources.yaml` | Bootstrap cluster | Installer import list | +| `48-installer-config.json` | Installer API | Installation request without credentials | + +The naming rule is a hard requirement: the `Cluster` and the `BaremetalCluster` are named exactly +`global`, and every other resource starts with `global-`. A resource without the prefix installs and +runs normally and is then invisible to DR — its machines are deleted at the first failover +([naming convention](https://docs.alauda.io/immutable-infra/1.0/global/install.html#common-prerequisites)). + +#### Build the registration ISO + +The bootstrap `MachineRegistration` registers the hosts against the bootstrap node. In DR it carries +the `global` auth-scope annotation — the only DR annotation allowed before handoff; never set +`baremetal.cluster.io/system-agent-server-url` on it. + +`global-primary/10-registration.yaml`: + +```yaml +apiVersion: elemental.cattle.io/v1beta1 +kind: MachineRegistration +metadata: + name: global-registration + namespace: cpaas-system + labels: + app.kubernetes.io/part-of: cluster-api-provider-baremetal + annotations: + # DR only: marks the global hosts for the handoff. Remove for a single cluster. + baremetal.cluster.io/system-agent-auth-scope: global +spec: + # ${System Information/...} are expanded by elemental-register from SMBIOS data, not by render. + machineName: "global-${System Information/UUID}" + machineInventoryLabels: + app.kubernetes.io/part-of: cluster-api-provider-baremetal + pool.baremetal.alauda.io/eligible: "true" + elemental.cattle.io/serial-number: "${System Information/Serial Number}" + machineInventoryAnnotations: + elemental.cattle.io/machine-uuid: "${System Information/UUID}" + config: + elemental: + install: + # Fail-closed install target, linked to the system disk WWN at the console. + device: /dev/elemental-install-target + eject-cd: true + reboot: true + snapshotter: + type: btrfs + maxSnaps: 4 + registration: + auth: tpm + # Hosts without a hardware TPM. With a hardware TPM on every host: + # emulate-tpm: false and remove emulated-tpm-seed. + emulate-tpm: true + emulated-tpm-seed: -1 +--- +apiVersion: elemental.cattle.io/v1beta1 +kind: SeedImage +metadata: + name: global-registration-iso + namespace: cpaas-system + labels: + app.kubernetes.io/part-of: cluster-api-provider-baremetal +spec: + type: iso + baseImage: ${BOOTSTRAP_REGISTRY_ADDRESS}/tkestack/baremetal-base-image-iso:${OS_IMAGE_TAG} + registrationRef: + apiVersion: elemental.cattle.io/v1beta1 + kind: MachineRegistration + name: global-registration + namespace: cpaas-system + targetPlatform: linux/amd64 + size: 20Gi + cleanupAfterMinutes: 120 + cloud-config: + stages: + boot: + - name: "Size COS_STATE for reprovisioning" + files: + - path: /etc/elemental/config.d/partitions.yaml + permissions: 0644 + content: | + install: + partitions: + state: + size: 20480 +``` + +```bash +# on the bootstrap cluster +render global-primary.env global-primary/10-registration.yaml > /tmp/out.yaml +grep -n '\${' /tmp/out.yaml | grep -v 'System Information/' && echo "unreplaced variables" || \ + kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" apply -f /tmp/out.yaml + +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system wait --for=condition=SeedImageReady=True \ + seedimage/global-registration-iso --timeout=30m +reason=$(kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get seedimage global-registration-iso \ + -o jsonpath='{.status.conditions[?(@.type=="SeedImageReady")].reason}') +iso_url=$(kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get seedimage global-registration-iso -o jsonpath='{.status.downloadURL}') +sum_url=$(kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get seedimage global-registration-iso -o jsonpath='{.status.checksumURL}') +echo "${reason} ${iso_url} ${sum_url}" # SeedImageBuildSuccess and two non-empty URLs + +curl -fL -k -o global-registration.iso "${iso_url}" +curl -fsL -k "${sum_url}"; echo; sha256sum global-registration.iso # the two digests must match +``` + +Do not use `status.state` as the gate; the controller does not fill it. The URLs are cleared when the +download lifetime (`cleanupAfterMinutes`) expires even though the condition stays `True` — download +within that window. Use `--cacert` with the bootstrap CA instead of `-k` when you have it at hand. + +#### Boot and register the hosts + +Per host, one host at a time when disks have to be cleaned +([fixed system disk workflow](https://docs.alauda.io/immutable-infra/1.0/how-to/configure-fixed-install-disk-bare-metal.html)): + +1. Confirm the clock gate for this host. +2. Mount `global-registration.iso` through BMC virtual media and boot from it. +3. In the Live ISO console, block the automatic installer and confirm the ISO is the expected one: + +```bash +# in the Live ISO console +systemctl stop elemental-register-install.service +systemctl mask elemental-register-install.service +systemctl reset-failed elemental-register-install.service +grep -n '/dev/elemental-install-target' /run/initramfs/live/livecd-cloud-config.yaml +``` + +4. Configure the planned production network unless DHCP hands out exactly the planned address — this + becomes the node's permanent network. Look up the real connection name first: + +```bash +# in the Live ISO console +nmcli connection show +nmcli connection modify "Wired connection 1" ipv4.method manual \ + ipv4.addresses 192.0.2.21/24 ipv4.gateway 192.0.2.1 ipv4.dns 192.0.2.53 ipv4.ignore-auto-dns yes +nmcli connection up "Wired connection 1" +ip -br addr; ip route +``` + +5. Identify the system disk by WWN, size, model and controller address — never by kernel name alone. + When the host is being reinstalled, clean the approved disks and verify that no `COS_*` label + remains, as in the product procedure. Then link the install target and start the installation: + +```bash +# in the Live ISO console +lsblk -d -e7 -o NAME,PATH,TYPE,SIZE,MODEL,SERIAL,WWN,HCTL +ls -l /dev/disk/by-id/wwn-* +test ! -e /dev/elemental-install-target +ln -s /dev/disk/by-id/wwn-0x5000000000000000 /dev/elemental-install-target # this host's system disk +test -b /dev/elemental-install-target && readlink -f /dev/elemental-install-target + +systemctl unmask elemental-register-install.service +systemctl reset-failed elemental-register-install.service +systemctl start elemental-register-install.service +journalctl -fu elemental-register-install.service +``` + +6. After the installation reboots the host, eject the virtual media, restore disk-first boot and + confirm the host does not return to the Live ISO. + +On the first host, confirm the `COS_STATE` size before the rest of the fleet installs: on the host +console, `lsblk -o NAME,SIZE,LABEL | grep COS_STATE` must show about 20G, not 8G. + +#### Record the inventories + +```bash +# on the bootstrap cluster +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get machineinventories.elemental.cattle.io \ + -o custom-columns='NAME:.metadata.name,SERIAL:.metadata.labels.elemental\.cattle\.io/serial-number,READY:.status.conditions[?(@.type=="Ready")].status,PLAN:.status.plan.secretRef.name' +``` + +Match each serial number with the host plan, write the six inventory names and the hostnames into +the cluster file, and load it again. Then check every inventory — `Ready=True`, a plan Secret, the +planned network, and a fresh storage observation: + +```bash +# on the bootstrap cluster +load_env global-primary.env +for inv in "${BM_CP_INVENTORY_1}" "${BM_CP_INVENTORY_2}" "${BM_CP_INVENTORY_3}" \ + "${BM_INFRA_INVENTORY_1}" "${BM_INFRA_INVENTORY_2}" "${BM_INFRA_INVENTORY_3}"; do + echo "== ${inv}" + kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get machineinventory.elemental.cattle.io "${inv}" -o json | jq '{ + ready: ([.status.conditions[]? | select(.type=="Ready") | .status] | first), + plan: .status.plan.secretRef.name, + observedAt: .status.observedStorage.observedAt, + network: .spec.observedNetwork}' +done +``` + +#### Prepare /var/cpaas on every inventory + +Declare the data disk while the inventory is still outside any pool. Find the disk on each inventory +— use only a device whose `systemRole` is `Data` and whose report is fresh — and write the host file. +An empty `status.observedStorage` means the host OS has no `elemental-storage-observer.service` and +cannot use managed storage; stop and obtain an observer-capable image pair +([Availability and Upgrade Boundary](https://docs.alauda.io/immutable-infra/1.0/how-to/manage-bare-metal-storage.html#availability-and-upgrade-boundary)). + +```bash +# on the bootstrap cluster +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get machineinventory.elemental.cattle.io "${BM_CP_INVENTORY_1}" -o json \ + | jq -r '.status.observedStorage.devices[] | + [.id,.kind,.systemRole,(.sizeBytes|tostring),(.filesystem.type//"-"),([.mounts[].path]|join(","))] | @tsv' +``` + +`global-primary/15-var-cpaas-storage.yaml` — the input of `storagectl`, never applied directly: + +```yaml +storage: + retryNonce: 0 + volumes: + - name: cpaas-data + source: + deviceID: "${BM_DATA_DISK_ID}" + minimumSize: ${BM_DATA_DISK_MIN_SIZE} + filesystem: + policy: InitializeIfBlank + type: xfs + mount: + path: /var/cpaas + required: true + options: + - noatime +``` + +`InitializeIfBlank` formats the disk only when it is objectively blank, and only with an approval bound +to this inventory. The identity that applies it needs `update` on the virtual +`machineinventories/storageinitialize` subresource; the first command below must print `yes`. Per +host: + +```bash +# on the bootstrap cluster, one host at a time +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" auth can-i update machineinventories.elemental.cattle.io --subresource=storageinitialize -n cpaas-system +export BM_NS=cpaas-system +export BM_STORAGE_FILE=/tmp/baremetal-storage.yaml +export BM_INVENTORY_FILE=/tmp/baremetal-machineinventory.json +export BM_PATCH_FILE=/tmp/baremetal-storage.patch.json +host_file=hosts/global-primary/cp-1.env + +set -a; . "./${host_file}"; set +a +render "${host_file}" global-primary/15-var-cpaas-storage.yaml > "${BM_STORAGE_FILE}" +"${BM_STORAGECTL}" hash --storage "${BM_STORAGE_FILE}" + +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n "${BM_NS}" get machineinventory.elemental.cattle.io "${BM_INVENTORY}" -o json > "${BM_INVENTORY_FILE}" +counter=$(( $(jq '.status.storage.lastConsumedInitializationApprovalCounter // 0' "${BM_INVENTORY_FILE}") + 1 )) +"${BM_STORAGECTL}" render-patch --inventory "${BM_INVENTORY_FILE}" \ + --storage "${BM_STORAGE_FILE}" --counter "${counter}" > "${BM_PATCH_FILE}" + +# inspect spec.storage, the device ID, the path and the approval in this output first +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n "${BM_NS}" patch machineinventory.elemental.cattle.io "${BM_INVENTORY}" \ + --type=merge --patch-file="${BM_PATCH_FILE}" --dry-run=server -o yaml +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n "${BM_NS}" patch machineinventory.elemental.cattle.io "${BM_INVENTORY}" \ + --type=merge --patch-file="${BM_PATCH_FILE}" + +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n "${BM_NS}" get machineinventory.elemental.cattle.io "${BM_INVENTORY}" -o json \ + | jq '.status.storage | {phase, conditions, operation}' +``` + +Expect `phase=Prepared`, `StoragePrepared=True/AllRequiredVolumesPrepared`, +`StorageActive=False/Inactive` and `operation=null` on all six inventories before you continue. On a +`resourceVersion` conflict, fetch the inventory again and regenerate the patch; never strip +`resourceVersion` from it. + +#### Create the pools + +`global-primary/20-pools.yaml`: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: MachineInventoryPool +metadata: + name: global-control-plane-pool + namespace: cpaas-system +spec: + clusterName: global + machineInventories: + - name: ${BM_CP_INVENTORY_1} + hostname: ${BM_CP_HOSTNAME_1} + - name: ${BM_CP_INVENTORY_2} + hostname: ${BM_CP_HOSTNAME_2} + - name: ${BM_CP_INVENTORY_3} + hostname: ${BM_CP_HOSTNAME_3} +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: MachineInventoryPool +metadata: + name: global-infra-pool + namespace: cpaas-system +spec: + clusterName: global + machineInventories: + - name: ${BM_INFRA_INVENTORY_1} + hostname: ${BM_INFRA_HOSTNAME_1} + - name: ${BM_INFRA_INVENTORY_2} + hostname: ${BM_INFRA_HOSTNAME_2} + - name: ${BM_INFRA_INVENTORY_3} + hostname: ${BM_INFRA_HOSTNAME_3} +``` + +```bash +# on the bootstrap cluster +apply_rendered global-primary.env global-primary/20-pools.yaml "${BOOTSTRAP_KUBECONFIG}" +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io +``` + +Both pools must report `Ready=True` with `Total=3` and `Available=3`. A member that is missing, not +`Ready`, without a plan Secret or with unprepared storage raises `unavailable` and shows up in the +`MembersValid` condition. + +#### Create the control plane + +The `KubeadmControlPlane` is the production profile of +[Complete KubeadmControlPlane Configuration](https://docs.alauda.io/immutable-infra/1.0/create-cluster/bare-metal.html#complete-kubeadmcontrolplane-configuration) +merged with the `global` requirements of Step 4 and the DR increments. Keep the Kubernetes images on +`cloud.alauda.io/alauda`: they are preloaded in the OS image, and any other repository produces image +references that do not exist on the host. In DR, the `BaremetalCluster` references the shared +encryption Secret and the provider injects `/etc/kubernetes/encryption-provider.conf` from it; do not +also add that file to `files`. + +`global-primary/30-control-plane.yaml`: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: BaremetalCluster +metadata: + name: global + namespace: cpaas-system +spec: + controlPlaneLoadBalancer: + type: Internal + host: "${CONTROL_PLANE_VIP}" + port: 6443 + vrid: ${BM_CONTROL_PLANE_VRID} + networkType: kube-ovn + # DR only: the shared encryption configuration. Remove for a single cluster. + encryptionProviderConfigRef: + name: global-encryption-provider-config +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: BaremetalMachineTemplate +metadata: + name: global-control-plane-template + namespace: cpaas-system +spec: + template: + spec: + machineInventoryPoolRef: + name: global-control-plane-pool +--- +apiVersion: controlplane.cluster.x-k8s.io/v1beta1 +kind: KubeadmControlPlane +metadata: + name: global-kcp + namespace: cpaas-system + annotations: + controlplane.cluster.x-k8s.io/skip-kube-proxy: "" +spec: + replicas: 3 + version: "${K8S_VERSION}" + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + machineTemplate: + nodeDrainTimeout: 1m + nodeDeletionTimeout: 5m + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: BaremetalMachineTemplate + name: global-control-plane-template + kubeadmConfigSpec: + format: cloud-config + users: + - name: boot + sudo: ALL=(ALL) NOPASSWD:ALL + shell: /bin/bash + sshAuthorizedKeys: + - "${BOOT_SSH_PUBLIC_KEY}" + files: + - path: /etc/kubernetes/admission/psa-config.yaml + owner: "root:root" + permissions: "0644" + content: | + apiVersion: apiserver.config.k8s.io/v1 + kind: AdmissionConfiguration + plugins: + - name: PodSecurity + configuration: + apiVersion: pod-security.admission.config.k8s.io/v1 + kind: PodSecurityConfiguration + defaults: + enforce: "privileged" + enforce-version: "latest" + audit: "baseline" + audit-version: "latest" + warn: "baseline" + warn-version: "latest" + exemptions: + usernames: [] + runtimeClasses: [] + namespaces: + - kube-system + - cpaas-system + - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json + owner: "root:root" + permissions: "0644" + content: | + { + "apiVersion": "kubelet.config.k8s.io/v1beta1", + "kind": "KubeletConfiguration", + "imagePullCredentialsVerificationPolicy": "NeverVerify", + "protectKernelDefaults": true, + "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt", + "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key", + "streamingConnectionIdleTimeout": "5m", + "clientCAFile": "/etc/kubernetes/pki/ca.crt" + } + - path: /etc/kubernetes/audit/policy.yaml + owner: "root:root" + permissions: "0644" + content: | + apiVersion: audit.k8s.io/v1 + kind: Policy + omitStages: + - "RequestReceived" + rules: + - level: None + users: + - system:kube-controller-manager + - system:kube-scheduler + - system:serviceaccount:kube-system:endpoint-controller + verbs: ["get", "update"] + namespaces: ["kube-system"] + resources: + - group: "" + resources: ["endpoints"] + - level: None + nonResourceURLs: + - /healthz* + - /version + - /swagger* + - level: None + resources: + - group: "" + resources: ["events"] + - level: None + verbs: ["get", "list", "watch"] + - level: None + resources: + - group: "coordination.k8s.io" + resources: ["leases"] + - level: None + resources: + - group: "authorization.k8s.io" + resources: ["subjectaccessreviews", "selfsubjectaccessreviews"] + - group: "authentication.k8s.io" + resources: ["tokenreviews"] + - level: Metadata + resources: + - group: "" + resources: ["secrets", "configmaps"] + - level: RequestResponse + resources: + - group: "" + - group: "apps" + - group: "rbac.authorization.k8s.io" + - group: "storage.k8s.io" + - group: "networking.k8s.io" + - level: Metadata + # DR only: the shared ServiceAccount signing key. Remove both entries for a single cluster. + - path: /etc/kubernetes/pki/sa.key + owner: root:root + permissions: "0600" + contentFrom: + secret: + name: global-sa-signing-key + key: sa.key + - path: /etc/kubernetes/pki/sa.pub + owner: root:root + permissions: "0644" + contentFrom: + secret: + name: global-sa-signing-key + key: sa.pub + postKubeadmCommands: + - chmod 600 /var/lib/kubelet/config.yaml + clusterConfiguration: + imageRepository: cloud.alauda.io/alauda + dns: + imageTag: "${DNS_IMAGE_TAG}" + etcd: + local: + imageTag: "${ETCD_IMAGE_TAG}" + # DR list, identical on both sides. Single cluster: CONTROL_PLANE_VIP and PLATFORM_HOST only. + serverCertSANs: + - "${PRIMARY_CLUSTER_VIP}" + - "${STANDBY_CLUSTER_VIP}" + - "${PLATFORM_HOST}" + - "etcd.kube-system" + apiServer: + extraArgs: + audit-log-format: json + audit-log-maxage: "30" + audit-log-maxbackup: "10" + audit-log-maxsize: "200" + profiling: "false" + audit-log-mode: batch + audit-log-path: /etc/kubernetes/audit/audit.log + audit-policy-file: /etc/kubernetes/audit/policy.yaml + tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384" + admission-control-config-file: /etc/kubernetes/admission/psa-config.yaml + tls-min-version: VersionTLS12 + kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt + # DR only: sign and verify ServiceAccount tokens with the shared key. + service-account-key-file: /etc/kubernetes/pki/sa.pub + service-account-signing-key-file: /etc/kubernetes/pki/sa.key + service-account-issuer: https://kubernetes.default.svc.cluster.local + api-audiences: https://kubernetes.default.svc.cluster.local + extraVolumes: + - name: vol-dir-0 + hostPath: /etc/kubernetes + mountPath: /etc/kubernetes + pathType: Directory + controllerManager: + extraArgs: + bind-address: "::" + profiling: "false" + tls-min-version: VersionTLS12 + # DR only. + service-account-private-key-file: /etc/kubernetes/pki/sa.key + scheduler: + extraArgs: + bind-address: "::" + tls-min-version: VersionTLS12 + profiling: "false" + initConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + node-labels: "kube-ovn/role=master" + joinConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + node-labels: "kube-ovn/role=master" +--- +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: global + namespace: cpaas-system + labels: + cluster-type: ProviderBaremetal + annotations: + capi.cpaas.io/resource-group-version: infrastructure.cluster.x-k8s.io/v1beta1 + capi.cpaas.io/resource-kind: BaremetalCluster + cpaas.io/kube-ovn-join-cidr: "${KUBE_OVN_JOIN_CIDR}" + cpaas.io/kube-ovn-version: "${KUBE_OVN_VERSION}" + # The bootstrap Registry; the installer rewrites it to the platform Registry after installation. + cpaas.io/registry-address: "${NODE_REGISTRY_ADDRESS}" + cpaas.io/sentry-deploy-type: Baremetal + cpaas.io/alb-address-type: ClusterAddress # required literal of the Bare Metal profile +spec: + clusterNetwork: + pods: + cidrBlocks: + - ${CLUSTER_CIDR} + services: + cidrBlocks: + - ${SERVICE_CIDR} + controlPlaneRef: + apiVersion: controlplane.cluster.x-k8s.io/v1beta1 + kind: KubeadmControlPlane + name: global-kcp + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: BaremetalCluster + name: global +``` + +Check before applying: three replicas; the endpoint is an IP; the three CIDRs overlap neither each +other nor the host, VIP or other-cluster networks; in DR, the SAN list matches the other side's. + +```bash +# on the bootstrap cluster +apply_rendered global-primary.env global-primary/30-control-plane.yaml "${BOOTSTRAP_KUBECONFIG}" +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io -w # Pending → Allocated → Reprovisioning → Running +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get clusters.cluster.x-k8s.io global +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get kubeadmcontrolplane global-kcp +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get baremetalcluster global \ + -o jsonpath='{.status.conditions[?(@.type=="EndpointReady")].status}'; echo +``` + +The control plane is ready when the `KubeadmControlPlane` reports `Ready: True` and the `Cluster` +reports `Phase: Provisioned`. Save the `global` kubeconfig as described in the common +[Kubeconfig convention](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#kubeconfig-convention) +— read it before the bootstrap cluster is removed: + +```bash +# on the bootstrap cluster +umask 077; mkdir -p /root/.kube/acp +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get secret global-kubeconfig -o jsonpath='{.data.value}' | base64 -d \ + > "${CLUSTER_KUBECONFIG}" +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get nodes -o wide +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n kube-system get pods -l app=alive -o wide +``` + +#### Add the infra pool + +Apply the `infra` pool after the control plane is `Ready`. Labels and the taint are declared on the +pool, never set by hand on nodes; the `worker` role label puts the nodes into the default worker +MachineConfigPool, without which they silently miss time zone and NTP configuration. + +`global-primary/40-infra-pool.yaml`: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: BaremetalMachineTemplate +metadata: + name: global-infra-template + namespace: cpaas-system +spec: + template: + spec: + machineInventoryPoolRef: + name: global-infra-pool +--- +apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 +kind: KubeadmConfigTemplate +metadata: + name: global-worker-bootstrap + namespace: cpaas-system +spec: + template: + spec: + users: + - name: boot + sudo: ALL=(ALL) NOPASSWD:ALL + shell: /bin/bash + sshAuthorizedKeys: + - "${BOOT_SSH_PUBLIC_KEY}" + files: + - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json + owner: "root:root" + permissions: "0644" + content: | + { + "apiVersion": "kubelet.config.k8s.io/v1beta1", + "kind": "KubeletConfiguration", + "imagePullCredentialsVerificationPolicy": "NeverVerify", + "protectKernelDefaults": true, + "staticPodPath": null, + "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt", + "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key", + "streamingConnectionIdleTimeout": "5m", + "clientCAFile": "/etc/kubernetes/pki/ca.crt" + } + joinConfiguration: + patches: + directory: /etc/kubernetes/patches +--- +apiVersion: cluster.x-k8s.io/v1beta1 +kind: MachineDeployment +metadata: + name: global-infra + namespace: cpaas-system +spec: + clusterName: global + replicas: 3 + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + maxUnavailable: 1 + selector: + matchLabels: {} + template: + metadata: + labels: + cluster.x-k8s.io/cluster-name: global + pool.name: global-infra + node-role.kubernetes.io/infra: "" + node-role.kubernetes.io/worker: "" + spec: + clusterName: global + version: "${K8S_VERSION}" + nodeDrainTimeout: 5m + nodeVolumeDetachTimeout: 5m + taints: + - key: node-role.kubernetes.io/infra + value: reserved + effect: NoSchedule + bootstrap: + configRef: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: global-worker-bootstrap + namespace: cpaas-system + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: BaremetalMachineTemplate + name: global-infra-template + namespace: cpaas-system +``` + +```bash +# on the bootstrap cluster +apply_rendered global-primary.env global-primary/40-infra-pool.yaml "${BOOTSTRAP_KUBECONFIG}" +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get machinedeployment global-infra +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get baremetalmachine -l cluster.x-k8s.io/cluster-name=global \ + -o custom-columns='NAME:.metadata.name,PHASE:.status.phase,INVENTORY:.status.machineInventoryRef.name' +# on global: labels and taints arrived on the nodes, not only on the Machines +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get nodes -L node-role.kubernetes.io/infra,node-role.kubernetes.io/worker +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key' +``` + +Continue when all six `BaremetalMachine` objects are `Running`, each on the inventory you planned for +it. + +#### Import the provider resources + +The import ConfigMap must exist **before** the installer request. It carries the Bare Metal and +Elemental owner objects that the handoff uses to enumerate the `global` machines; without it the +handoff runs with an empty target list (Step 7 of +[Installing the global Cluster](https://docs.alauda.io/immutable-infra/1.0/global/install.html)). It +must not import plan Secrets, `KubeadmConfig` objects, bootstrap data Secrets, `SeedImage` objects or +the registration token Secret: imported too early, a plan Secret still points at the bootstrap owner +UID and is garbage-collected. All six `global` inventories are listed, so the `infra` hosts are handed +off too. + +`global-primary/45-import-extra-resources.yaml`: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: dcs-import-extra-resources + namespace: cpaas-system +data: + resources.yaml: | + resources: + - resource: "customresourcedefinitions.apiextensions.k8s.io" + names: + - baremetalclusters.infrastructure.cluster.x-k8s.io + - baremetalmachines.infrastructure.cluster.x-k8s.io + - baremetalmachinetemplates.infrastructure.cluster.x-k8s.io + - machineinventorypools.infrastructure.cluster.x-k8s.io + - machineinventories.elemental.cattle.io + - machineregistrations.elemental.cattle.io + - seedimages.elemental.cattle.io + method: kubectl + - resource: "baremetalclusters.infrastructure.cluster.x-k8s.io" + names: ["global"] + etcdKeyBase: "/registry/infrastructure.cluster.x-k8s.io/baremetalclusters/cpaas-system/" + method: etcdctl + - resource: "baremetalmachinetemplates.infrastructure.cluster.x-k8s.io" + etcdKeyBase: "/registry/infrastructure.cluster.x-k8s.io/baremetalmachinetemplates/cpaas-system/" + method: etcdctl + - resource: "baremetalmachines.infrastructure.cluster.x-k8s.io" + etcdKeyBase: "/registry/infrastructure.cluster.x-k8s.io/baremetalmachines/cpaas-system/" + method: etcdctl + - resource: "machineinventorypools.infrastructure.cluster.x-k8s.io" + etcdKeyBase: "/registry/infrastructure.cluster.x-k8s.io/machineinventorypools/cpaas-system/" + method: etcdctl + - resource: "machineinventories.elemental.cattle.io" + names: + - "${BM_CP_INVENTORY_1}" + - "${BM_CP_INVENTORY_2}" + - "${BM_CP_INVENTORY_3}" + - "${BM_INFRA_INVENTORY_1}" + - "${BM_INFRA_INVENTORY_2}" + - "${BM_INFRA_INVENTORY_3}" + etcdKeyBase: "/registry/elemental.cattle.io/machineinventories/cpaas-system/" + method: etcdctl + - resource: "machineregistrations.elemental.cattle.io" + names: ["global-registration"] + etcdKeyBase: "/registry/elemental.cattle.io/machineregistrations/cpaas-system/" + method: etcdctl + # DR only: the Secret referenced by BaremetalCluster.spec.encryptionProviderConfigRef. + # Remove this entry for a single cluster. + - resource: "secrets" + names: + - "global-encryption-provider-config" + method: kubectl +``` + +```bash +# on the bootstrap cluster: the live inventory references must equal the six names in the file +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get baremetalmachine -l cluster.x-k8s.io/cluster-name=global \ + -o jsonpath='{range .items[*]}{.status.machineInventoryRef.name}{"\n"}{end}' | sort +apply_rendered global-primary.env global-primary/45-import-extra-resources.yaml "${BOOTSTRAP_KUBECONFIG}" +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get cm dcs-import-extra-resources -o yaml +# DR only: must print global-encryption-provider-config, the Secret name listed in the file +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get baremetalcluster global \ + -o jsonpath='{.spec.encryptionProviderConfigRef.name}'; echo +``` + +In DR, an empty result means the installed `BaremetalCluster` CRD dropped the reference; stop, because +the control plane is then running without the shared encryption configuration. + +#### Trigger the platform installation + +The request file is a template without credentials. The passwords are read at a prompt, injected with +`jq` into a `0600` working copy, and the copy is deleted right after the request. On Bare Metal, +`REGISTRY_DOMAIN` is `:11443` and `registry.externalAddress` stays unset, so the +installer deploys the platform Registry. `console.host` is the local VIP, never the domain; the domain +goes into `console.globalHost`. + +`global-primary/48-installer-config.json`: + +```json +{ + "basic": { + "username": "admin@cpaas.io", + "password": "" + }, + "registry": { + "domain": "${REGISTRY_DOMAIN}", + "username": "", + "password": "" + }, + "console": { + "host": [ + "${CONTROL_PLANE_VIP}" + ], + "globalHost": "${PLATFORM_HOST}", + "httpPort": 80, + "httpsPort": 443, + "cert": { + "selfSigned": {} + } + }, + "cluster": { + "clusterCIDR": "${CLUSTER_CIDR}", + "serviceCIDR": "${SERVICE_CIDR}", + "features": { + "ha": { + "vip": "${CONTROL_PLANE_VIP}", + "vport": 6443, + "isThirdParty": true + } + } + }, + "product": [ + "base", + "acp" + ], + "deployMode": "normal", + "hostIP": "${HOST_IP}" +} +``` + +DR: both sides must serve the **same trusted third-party certificate** for `PLATFORM_HOST`; two +unrelated self-signed certificates break the hosts and the standby after a failover. Replace +`"selfSigned": {}` with the `thirdParty` block that carries the base64 full chain and private key, as +the Third-Party Console Certificates note of Step 8 in +[Installing the global Cluster](https://docs.alauda.io/immutable-infra/1.0/global/install.html) +describes, and inject the private key at submit time the same way as the passwords — never into the +template. Its SAN is the platform domain; add a VIP only when clients really use platform HTTPS through +that VIP. Use the same platform administrator and Registry credentials on both sides, as +[Global Cluster Disaster Recovery](https://docs.alauda.io/container_platform/4.4/install/global_dr.html) +requires. + +```bash +# on the bootstrap node +load_env global-primary.env +export INSTALLER_IP=$(kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" get pods -n cpaas-system -l service_name=cpaas-installer \ + -o jsonpath='{.items[0].status.podIP}') +umask 077 +mkdir -p /root/yamls +read -rsp 'Platform admin password: ' pw; echo +export ADMIN_PASSWORD_B64="$(printf '%s' "${pw}" | base64 -w0)" +read -rp 'Registry username: ' REGISTRY_USERNAME; export REGISTRY_USERNAME +read -rsp 'Registry password: ' pw; echo +export REGISTRY_PASSWORD_B64="$(printf '%s' "${pw}" | base64 -w0)" +unset pw + +render global-primary.env global-primary/48-installer-config.json | jq ' + .basic.password = env.ADMIN_PASSWORD_B64 + | .registry.username = env.REGISTRY_USERNAME + | .registry.password = env.REGISTRY_PASSWORD_B64' > /root/yamls/installer-config.json +unset ADMIN_PASSWORD_B64 REGISTRY_USERNAME REGISTRY_PASSWORD_B64 + +grep -n '\${' /root/yamls/installer-config.json && echo "unreplaced variables" || \ + curl -k -X POST "http://${INSTALLER_IP}:8080/cpaas-installer/api/config/dcs" \ + -H 'Content-Type: application/json' -d @/root/yamls/installer-config.json +rm -f /root/yamls/installer-config.json +``` + +#### Monitor and verify + +A `global` installation takes 30 to 60 minutes (Step 9): + +```bash +# on the bootstrap node +curl "http://${INSTALLER_IP}:8080/cpaas-installer/api/progress" +tail -f /var/cpaas/data/installer.log +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get nodes +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get clusters.platform.tkestack.io global -o jsonpath='{.status.phase}'; echo +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get pods -n cpaas-system +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get clustermodule global +``` + +The installation is complete when the progress API reports `status: Success` and `type: Complete`, +all six nodes are `Ready`, the critical `cpaas-system` Pods are `Running` or `Completed`, and +`ClusterModule/global` reports the base module healthy. Then check the Bare Metal specifics: + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get nodes \ + -o custom-columns='NAME:.metadata.name,PROVIDER_ID:.spec.providerID,IP:.status.addresses[?(@.type=="InternalIP")].address' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get cluster.cluster.x-k8s.io global \ + -o jsonpath='{.metadata.annotations.cpaas\.io/registry-address}'; echo # now REGISTRY_DOMAIN + +# DR only: the imported encryption Secret matches the shared file (compare digests, never print it) +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get secret global-encryption-provider-config \ + -o jsonpath='{.data.encryption-provider\.conf}' | base64 -d | sha256sum +sha256sum "${ENCRYPTION_PROVIDER_CONF}" +# DR only: the KubeadmControlPlane reads the signing key from this Secret whenever it builds a new +# control-plane node, so it must exist on global as well +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get secret global-sa-signing-key +``` + +Expect six nodes, every `providerID` `baremetal:///`, and the planned node IPs. + +The product's import list does not include `global-sa-signing-key`, and the product documentation does +not say how `global` obtains it after handoff. If it is missing on `global`, create it there from the +shared files with the same `kubectl create secret` command as on the bootstrap cluster, while the +bootstrap node still holds them: the `KubeadmControlPlane` cannot build bootstrap data for a new +control-plane node without it. DR only — on one control-plane node of each side, the digests must equal +those of the shared files and the flags must be identical on both sides: + +```bash +# on a control-plane node of each global side +sudo sha256sum /etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.pub /etc/kubernetes/encryption-provider.conf +sudo grep -E 'service-account-(issuer|key-file|signing-key-file)|api-audiences|encryption-provider-config' \ + /etc/kubernetes/manifests/kube-apiserver.yaml +sudo grep service-account-private-key-file /etc/kubernetes/manifests/kube-controller-manager.yaml +``` + +#### Prepare global as the management cluster + +The installation imports only the Bare Metal CRDs. It does not install the providers on `global`, nor +the OS images, nor the catalog entry; without this step `global` cannot replace its own nodes once the +bootstrap node is gone, and cannot create workload clusters +([Prepare the global Cluster as a Management Cluster](https://docs.alauda.io/immutable-infra/1.0/global/install.html#prepare-global-as-management-cluster)). +Run it on every `global` side **while the platform domain resolves to that side**: the Registry +address in `ProductBase` is `:11443` on both sides, so the domain decides which Registry +receives the push, and the push must authenticate with the `registry-admin` credential of that same +side. + +```bash +# on the bootstrap node of this side +load_env global-primary.env +getent hosts "${PLATFORM_HOST}" # must print this side's CONTROL_PLANE_VIP +GLOBAL_REGISTRY_ADDRESS="$(kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" \ + get productbase.product.alauda.io base -o jsonpath='{.spec.registry.address}')" +GLOBAL_REGISTRY_PASSWORD="$(kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system \ + get secret registry-admin -o jsonpath='{.data.password}' | base64 -d)" +``` + +1. **Upload the packages and enable label synchronization on this side** — the common + [Install the platform plugins](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#install-the-platform-plugins) + (Upload the packages, Enable label synchronization). The upload carries the two provider packages + and, for DR, `etcd-sync`. Replacement `infra` nodes and every workload node get their labels from + this side's controller from now on. + +2. **Install both providers on `global`** as cluster plugins + ([Cluster Plugin, Installing via YAML](https://docs.alauda.io/container_platform/4.4/extend/cluster_plugin.html#installing-via-yaml)). + The product documentation does not list the plugin's configuration keys, so read the Bare Metal + provider's `ModuleConfig` first and set through it the same system-agent values as on this side's + bootstrap release, plus the final certificate key — the check below is the gate: + `elemental.systemAgent.splitAuthEnabled` = `BM_SPLIT_AUTH_ENABLED`, + `elemental.systemAgent.sharedAuthReadOnly` = `BM_SHARED_AUTH_READ_ONLY`, and + `elemental.tls.caCertSecretKey: tls.crt`. On `global`, `dex.tls` comes from the installer and has no + `ca.crt` key; with the default key `elemental-operator` never starts + (`references non-existent secret key: ca.crt`). Then verify the effective values: + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get deploy \ + cluster-api-provider-baremetal-manager elemental-operator +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get secret dex.tls -o json \ + | jq -r '.data | keys | join(",")' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get apprelease -A -o json | \ + jq -e --argjson split "${BM_SPLIT_AUTH_ENABLED}" --argjson ro "${BM_SHARED_AUTH_READ_ONLY}" ' + [.items[] | select(any(.spec.source.charts[]?; .name | test("baremetal")))] | + length == 1 and + (.[0].spec.values.elemental.systemAgent.splitAuthEnabled // false) == $split and + (.[0].spec.values.elemental.systemAgent.sharedAuthReadOnly // false) == $ro and + .[0].spec.values.elemental.tls.caCertSecretKey == "tls.crt"' +``` + +3. **Push both OS images to this side's Registry.** Workload `SeedImage` builds and every later + reprovision of a `global` node pull from it; until then an ISO build fails with + `MANIFEST_UNKNOWN`. + +```bash +# on the bootstrap node +for repo in baremetal-base-image baremetal-base-image-iso; do + ctr -n bare-metal-os images push --local --skip-verify \ + -u "admin:${GLOBAL_REGISTRY_PASSWORD}" \ + "${GLOBAL_REGISTRY_ADDRESS}/tkestack/${repo}:${OS_IMAGE_TAG}" "${repo}:${OS_IMAGE_TAG}" + curl -sk -u "admin:${GLOBAL_REGISTRY_PASSWORD}" \ + "https://${GLOBAL_REGISTRY_ADDRESS}/v2/tkestack/${repo}/tags/list" +done +unset GLOBAL_REGISTRY_PASSWORD +``` + +Each response must list `OS_IMAGE_TAG`. Then remove the local copies with +`ctr -n bare-metal-os images rm`. + +4. **Add the catalog entry.** `global` does not inherit the bootstrap entry. Patch, do not replace, + so that entries added later for upgrades survive: + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system patch configmap elemental-image-catalog --type merge \ + -p "{\"data\":{\"${K8S_VERSION}\":\"${GLOBAL_REGISTRY_ADDRESS}/tkestack/baremetal-base-image:${OS_IMAGE_TAG}\"}}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get configmap elemental-image-catalog -o jsonpath='{.data}'; echo +``` + +#### Retire the bootstrap cluster + +Single cluster: retire it after this gate. DR: only after the handoff gate has passed on **both** +sides in [Add disaster recovery](#add-disaster-recovery). Do not retire it because the installer +reports success or the handoff Job is `Complete` — every imported host must have reported to the final +cluster first +([Decommission the Bootstrap Cluster](https://docs.alauda.io/immutable-infra/1.0/global/install.html#decommission-bootstrap-cluster)): + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get configmap baremetal-system-agent-handoff \ + -o jsonpath='{.data.ready}'; echo # true +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get machineinventories.elemental.cattle.io \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.plan.state}{"\n"}{end}' # all Applied +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get serviceaccount,secret,role,rolebinding \ + | grep -E 'baremetal-(global-)?system-agent' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system \ + get rolebinding baremetal-global-system-agent-handoff-bridge # NotFound +``` + +Never patch `data.ready` by hand. With split authentication disabled (single cluster) `global` +carries only the shared `baremetal-system-agent` identity; with it enabled it also carries +`baremetal-global-system-agent`, and the two Roles never name the same plan Secret. Then remove only +the bootstrap cluster `minialauda` and its KIND container network from the bootstrap node; the product +documentation names these two objects but gives no removal command, so obtain it from Alauda technical +support rather than improvising one. **Never +delete the `Cluster`, `KubeadmControlPlane` or Bare Metal objects as cleanup** — they now own the live +`global` nodes, and deleting them destroys the cluster you just installed. + +### Add disaster recovery + +#### Install order + +Two bootstrap nodes, one per side. The order is fixed, and what switches is only the platform domain — +each cluster is always reached through its own VIP +([Optional Disaster Recovery Deployment](https://docs.alauda.io/immutable-infra/1.0/global/install.html#optional-disaster-recovery-deployment)). +The stages are those of the common +[delivery flow](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#delivery-flow): + +| Common stage | Platform domain resolves to | Bare Metal action | +| ---: | --- | --- | +| 4 | — | [Prepare Bare Metal](#prepare-bare-metal), including [Generate the shared DR material](#generate-the-shared-dr-material); confirm the bidirectional DR network | +| 7 | `PRIMARY_CLUSTER_VIP` | Primary: [Prepare the bootstrap cluster](#prepare-the-bootstrap-cluster) and [Install the global cluster](#install-the-global-cluster) up to [Prepare global as the management cluster](#prepare-global-as-the-management-cluster) | +| 8 | `PRIMARY_CLUSTER_VIP` | Common plugins and machine configuration on the primary | +| 9 | `STANDBY_CLUSTER_VIP` | Standby: [Install the standby](#install-the-standby) | +| 10 | `STANDBY_CLUSTER_VIP` | Common plugins and machine configuration on the standby | +| 11 | `PRIMARY_CLUSTER_VIP` | [Handoff gate](#verify-the-handoff-on-both-sides), [rule ConfigMap](#apply-the-rule-configmap-on-both-sides), [etcd-sync](#install-etcd-sync-on-the-standby), [verification](#verify-synchronization) | +| 12 | `PRIMARY_CLUSTER_VIP` | Retire both bootstrap clusters | + +The common delivery flow explains why the domain moves. After every switch, confirm with +`getent hosts "${PLATFORM_HOST}"` on the bootstrap node and on a node of the side — a cached record +silently installs half a side against the other one. + +#### Install the standby + +On the standby bootstrap node, with the shared files in `/root/global-dr/`: + +```bash +# on the standby bootstrap node, in the delivery directory +cp -r global-primary global-standby # unchanged copy; every difference is in global-standby.env +load_env global-standby.env +getent hosts "${PLATFORM_HOST}" # must print STANDBY_CLUSTER_VIP +``` + +Run [Prepare the bootstrap cluster](#prepare-the-bootstrap-cluster) and every step of +[Install the global cluster](#install-the-global-cluster) with `global-standby.env`, the +`global-standby/` files and the `hosts/global-standby/` files. The differences come only from the +cluster file: `HOST_IP` and `NODE_REGISTRY_ADDRESS` of the standby bootstrap node, `CONTROL_PLANE_VIP` += `STANDBY_CLUSTER_VIP` (in the `BaremetalCluster`, `handoffHook.controlPlaneVIP`, `console.host` and +`cluster.features.ha.vip`), its own `vrid`, inventories and hostnames, and +`BM_SHARED_AUTH_READ_ONLY='true'`. `REGISTRY_DOMAIN` stays `:11443`, and `INSTALLER_IP` +is read from the standby bootstrap cluster, never reused from the primary. + +#### Verify the handoff on both sides + +Once stage 10 is complete on the standby, switch the platform domain back to `PRIMARY_CLUSTER_VIP` and +confirm it; the checks below do not depend on the domain. Run them from the delivery host, with SSH +access to the `global` hosts and four kubeconfigs: both `global` ones and a copy of each side's +`BOOTSTRAP_KUBECONFIG`. Keep shell tracing off, because the files read on the hosts contain a bearer +token ([Verify Before Decommissioning Either Bootstrap Cluster](https://docs.alauda.io/immutable-infra/1.0/global/install.html#bare-metal-dr-verification)). + +```bash +# on the delivery host +load_env global-primary.env +set +x +export PRIMARY_GLOBAL_KUBECONFIG=/root/.kube/acp/global-primary.kubeconfig +export STANDBY_GLOBAL_KUBECONFIG=/root/.kube/acp/global-standby.kubeconfig +export PRIMARY_BOOTSTRAP_KUBECONFIG=/root/.kube/acp/bootstrap-primary.kubeconfig +export STANDBY_BOOTSTRAP_KUBECONFIG=/root/.kube/acp/bootstrap-standby.kubeconfig +export PRIMARY_GLOBAL_HOSTS="192.0.2.21 192.0.2.22 192.0.2.23 192.0.2.24 192.0.2.25 192.0.2.26" +export STANDBY_GLOBAL_HOSTS="203.0.113.21 203.0.113.22 203.0.113.23 203.0.113.24 203.0.113.25 203.0.113.26" +export PRIMARY_GLOBAL_REGISTRATION_NAME=global-registration +export STANDBY_GLOBAL_REGISTRATION_NAME=global-registration +``` + +First the Registry propagation. Both sides must carry the identical `:11443` +everywhere, which is what lets synchronization replicate these keys without overwriting anything: + +```bash +# on the delivery host +verify_baremetal_global_registry() { + kubeconfig=$1 + expected_registry=$2 + + test "$(kubectl --kubeconfig "${kubeconfig}" \ + get productbase.product.alauda.io base \ + -o jsonpath='{.spec.registry.address}')" = "${expected_registry}" + test "$(kubectl --kubeconfig "${kubeconfig}" \ + get cluster.platform.tkestack.io global \ + -o jsonpath='{.metadata.annotations.cpaas\.io/registry-address}')" = \ + "${expected_registry}" + test "$(kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get cluster.cluster.x-k8s.io global \ + -o jsonpath='{.metadata.annotations.cpaas\.io/registry-address}')" = \ + "${expected_registry}" + test "$(kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get secret public-registry-credential -o jsonpath='{.data.registry}' | \ + base64 -d)" = "${expected_registry}" + + if kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get secret global-registry-auth >/dev/null 2>&1; then + kubectl --kubeconfig "${kubeconfig}" -n cpaas-system \ + get secret global-registry-auth -o jsonpath='{.data.\.dockerconfigjson}' | \ + base64 -d | jq -e --arg registry "${expected_registry}" \ + '.auths[$registry] != null' >/dev/null + fi + + kubectl --kubeconfig "${kubeconfig}" get apprelease -A -o json | \ + jq -e --arg registry "${expected_registry}" ' + all(.items[]; + .spec.source.repoURL == $registry and + .spec.values.global.registry.address == $registry) + ' >/dev/null +} + +verify_baremetal_global_registry "${PRIMARY_GLOBAL_KUBECONFIG}" "${REGISTRY_DOMAIN}" && echo "primary ok" +verify_baremetal_global_registry "${STANDBY_GLOBAL_KUBECONFIG}" "${REGISTRY_DOMAIN}" && echo "standby ok" +``` + +Then the complete handoff gate. Save the three functions `verify_global_auth_scope`, +`verify_baremetal_handoff` and `verify_global_local_system_agent_rbac` from the product section +verbatim as `dr-gate.sh` — per host they check that the system agent uses the local VIP, the local +token and CA, its own plan Secret and strict TLS, and that the two permission sets do not overlap — +then run: + +```bash +# on the delivery host +( set -euo pipefail + . ./dr-gate.sh + verify_baremetal_handoff \ + "${PRIMARY_BOOTSTRAP_KUBECONFIG}" "${PRIMARY_GLOBAL_KUBECONFIG}" \ + "${PRIMARY_CLUSTER_VIP}" \ + "${PRIMARY_GLOBAL_HOSTS}" "${PRIMARY_GLOBAL_REGISTRATION_NAME}" + verify_baremetal_handoff \ + "${STANDBY_BOOTSTRAP_KUBECONFIG}" "${STANDBY_GLOBAL_KUBECONFIG}" \ + "${STANDBY_CLUSTER_VIP}" \ + "${STANDBY_GLOBAL_HOSTS}" "${STANDBY_GLOBAL_REGISTRATION_NAME}" + verify_global_local_system_agent_rbac "${PRIMARY_GLOBAL_KUBECONFIG}" + verify_global_local_system_agent_rbac "${STANDBY_GLOBAL_KUBECONFIG}" + echo "handoff gate passed" ) +``` + +The script logs in to the hosts as `root`; provide an SSH identity that can read the agent files on +every `global` host. Do not reduce acceptance to `data.ready`: a token or CA mismatch, a stale +bootstrap endpoint in `/oem/elemental-system-agent.yaml`, a failed plan probe or a wrong permission +boundary means the handoff is incomplete even when `ready` is `true`. Do not create the shared +`baremetal-system-agent` bundle by hand to satisfy the gate — it appears with the first workload +registration and is then synchronized. + +#### Apply the rule ConfigMap on both sides + +Create `baremetal-dr-rules` on **both** sides **before** installing `etcd-sync`: installing the plugin with the +active VIP and token set opens the sync path at once, and a sync without this rule source gives the +standby hosts the primary's plans. The ConfigMap lists every `global` plan Secret of both sides; read +them now — the handoff created them, so names collected earlier are stale +([Synchronization Rules](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#synchronization-rules)). + +```bash +# on the delivery host +mkdir -p global-dr +for kc in "${PRIMARY_GLOBAL_KUBECONFIG}" "${STANDBY_GLOBAL_KUBECONFIG}"; do + kubectl --kubeconfig "${kc}" -n cpaas-system get machineinventories.elemental.cattle.io -o json | \ + jq -r '.items[] + | select(.metadata.annotations["baremetal.cluster.io/system-agent-auth-scope"] == "global") + | "/registry/secrets/cpaas-system/" + .status.plan.secretRef.name' +done | sort -u > global-dr/ignore-equal.txt +wc -l < global-dr/ignore-equal.txt # 12: six hosts per side + +for kc in "${PRIMARY_GLOBAL_KUBECONFIG}" "${STANDBY_GLOBAL_KUBECONFIG}"; do + kubectl --kubeconfig "${kc}" -n cpaas-system create configmap baremetal-dr-rules \ + --from-file=ignore-equal.txt=global-dr/ignore-equal.txt --dry-run=client -o yaml | \ + kubectl label --local -f - etcd-sync.cpaas.io/enabled=true -o yaml | \ + kubectl annotate --local -f - etcd-sync.cpaas.io/weight=100 -o yaml | \ + kubectl --kubeconfig "${kc}" apply -f - +done +``` + +The result is the product template: `ConfigMap/cpaas-system/baremetal-dr-rules` with the label +`etcd-sync.cpaas.io/enabled: "true"`, the annotation `etcd-sync.cpaas.io/weight: "100"` and one +`ignore-equal.txt` line per `global` plan Secret. The plugin chart supplies every other Bare Metal rule; +do not duplicate them. Update the ConfigMap on both sides whenever a `global` plan Secret is added or +replaced — for example after a `global` host is registered again. + +#### Install etcd-sync on the standby + +Install the `etcd-sync` plugin (ACP 4.4.0 or later; the package is `etcd-sync`, not `global-etcd-sync`) +on the **standby only** — never on both sides at once +([Install etcd-sync](https://docs.alauda.io/immutable-infra/1.0/global/install.html#install-etcd-sync)). + +Create the token Secret on the standby from the active cluster's `k8sadmin` token. The pipe keeps the +token off the screen, the command line and the shell history: + +```bash +# on the delivery host: from the primary global into the standby global +kubectl --kubeconfig "${PRIMARY_GLOBAL_KUBECONFIG}" -n cpaas-system get secret k8sadmin \ + -o jsonpath='{.data.token}' | base64 -d | \ + kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system \ + create secret generic etcd-sync-active-cluster-token --from-file=token=/dev/stdin +``` + +Install the plugin on the standby `global` — Web Console through the standby VIP, Marketplace → +Cluster Plugins, **etcd Synchronizer** — with: + +| Parameter | Value | +| --- | --- | +| Active Global Cluster VIP | `PRIMARY_CLUSTER_VIP` | +| Active Global Cluster ETCD Endpoints | `https://:2379`, unless a load balancer forwards `2379` | +| Standby Cluster ETCD Endpoints | Default | +| Active Global Cluster Token Secret | `etcd-sync-active-cluster-token` | +| Data Check Interval | Default | +| Print detail logs | Disabled | + +```bash +# on the standby global (from the delivery host) +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system get configmap baremetal-dr-rules \ + -o jsonpath='{.metadata.annotations.etcd-sync\.cpaas\.io/status}'; echo # accepted +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system get configmap etcd-sync-rule-snapshot \ + -o jsonpath='{.data.snapshot\.json}'; echo # contains the source +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system get job etcd-sync-bootstrap +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system get po -l app=etcd-sync +``` + +A `rejected` rule source is not active: fix it (data keys, syntax, the `message` annotation) before +the mirror is allowed to write. + +#### Verify synchronization + +On a standby control-plane node, trigger the monitor check; it answers HTTP `425` until a check is +allowed. Require zero missed and zero surplus keys +([Enable etcd Synchronization](https://docs.alauda.io/container_platform/4.4/install/global_dr.html#etcd_sync)): + +```bash +# on a control-plane node of the standby global +mirror_svc=$(sudo kubectl get svc -n cpaas-system etcd-sync-monitor -o jsonpath='{.spec.clusterIP}') +until curl -fsg "http://${mirror_svc}/check"; do sleep 30; done +``` + +The shared workload system-agent bundle must be absent on both sides or complete on both sides: + +```bash +# on the delivery host +for kc in "${PRIMARY_GLOBAL_KUBECONFIG}" "${STANDBY_GLOBAL_KUBECONFIG}"; do + kubectl --kubeconfig "${kc}" -n cpaas-system get --ignore-not-found -o name \ + serviceaccount/baremetal-system-agent secret/baremetal-system-agent-token \ + role/baremetal-system-agent rolebinding/baremetal-system-agent | wc -l +done # 0 and 0, or 4 and 4 +``` + +Restart the Pods that load DR and endpoint configuration, on a primary and on a standby control-plane +node: + +```bash +# on a control-plane node of each global side +sudo kubectl delete po -n cpaas-system -l 'service_name in (alertmanager,vmselect,vminsert)' +sudo kubectl delete po -n cpaas-system -l service_name=cpaas-elasticsearch +sudo kubectl delete po -n cpaas-system -l service_name=cluster-transformer +``` + +Then retire both bootstrap clusters as in [Retire the bootstrap cluster](#retire-the-bootstrap-cluster). +AppRelease readiness alone is not the DR acceptance — the failover drill is +([Failover](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#failover-procedure)). +Before any later uninstall of `etcd-sync` — failover, failback, a DR-aware upgrade — the check above +must report zero differences: an inconsistent standby can resolve owner references wrongly and delete +workload `Machine` objects, which on Bare Metal runs `clean` plans on real hosts. + +#### Register a replacement global host after handoff + +The bootstrap `global` ISO points at a bootstrap endpoint that no longer exists. To add or replace a +`global` host later, create a new registration and `SeedImage` on the **active** `global`, named with +the `global-` prefix, with the local identity annotations +([Endpoint and Identity](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#endpoint-and-identity)): + +```yaml +metadata: + annotations: + baremetal.cluster.io/system-agent-auth-scope: global + baremetal.cluster.io/system-agent-server-url: https://${CONTROL_PLANE_VIP}:6443 + baremetal.cluster.io/system-agent-endpoint-mode: direct-apiserver +``` + +Point its `SeedImage.spec.baseImage` at `REGISTRY_DOMAIN`, prepare `/var/cpaas` on the new inventory, +add it to the pool, and add its plan Secret to `baremetal-dr-rules` on both sides. Never synchronize a +primary `global` registration to the standby. + +#### DR checks + +- Both sides passed [Monitor and verify](#monitor-and-verify) and the handoff gate. +- The encryption configuration, the signing key files and the etcd SAN list are identical on both + sides. +- `baremetal-dr-rules` is `accepted` on the standby and present on the primary. +- The monitor reports zero missed and zero surplus keys. +- A failover drill has been run, and after it the + [Bare Metal acceptance](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#bare-metal-dr-model) + holds: workload `Cluster`, `Machine` and `BaremetalMachine` objects intact; a workload plan reaches + `Applied`; worker scale-out and scale-in work; a workload ISO still registers a host; `global` hosts + still use their local VIP, token and plan Secrets. + +### Create a workload cluster + +#### Before you start + +Create workload clusters on the **active** `global` (in DR, the primary). Check on `global`: + +```bash +# on global +load_env wl-prod.env +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get configmap elemental-image-catalog \ + -o jsonpath='{.data}'; echo # contains K8S_VERSION +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get cluster.cluster.x-k8s.io global \ + -o jsonpath='{.metadata.annotations.cpaas\.io/registry-address}'; echo # REGISTRY_DOMAIN +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get deploy capi-controller-manager \ + -o jsonpath='{.spec.template.spec.containers[0].args}'; echo # label sync enabled +``` + +- **Name.** `CLUSTER_NAME` must not be `global` + ([Creating Clusters on Bare Metal](https://docs.alauda.io/immutable-infra/1.0/create-cluster/bare-metal.html#cluster-creation-workflow)). + Do not start it with `global-` either: DR identifies the resources of the `global` pair by that prefix + ([naming convention](https://docs.alauda.io/immutable-infra/1.0/global/install.html#common-prerequisites)), + and every workload resource name starts with `CLUSTER_NAME`. +- **Registry.** A cluster is bound to its Registry at creation and cannot move later. The default is + the `global` Registry; decide before applying + ([Choose the Image Registry for a Workload Cluster](https://docs.alauda.io/immutable-infra/1.0/how-to/workload-cluster-image-registry.html)). +- **Network.** Workload hosts reach the platform through the domain on `443` (registration and plan + polling on `/kubernetes/global`) and the Registry on `11443`; both `global` sides reach the workload + VIP on `6443`. A workload ISO is built on `global` with the platform domain, so it stays valid + across a failover. +- **Hosts.** The same preparation as for `global`: [Prepare every host](#prepare-every-host) and + [Pass the clock gate](#pass-the-clock-gate). + +#### Workload manifests + +The files differ from the `global` ones by the cluster name prefix, the platform Registry as image +source, no bootstrap or DR settings, and the workload timeouts of the product page. + +`wl-prod/10-registration.yaml`: + +```yaml +apiVersion: elemental.cattle.io/v1beta1 +kind: MachineRegistration +metadata: + name: ${CLUSTER_NAME}-registration + namespace: cpaas-system + labels: + app.kubernetes.io/part-of: cluster-api-provider-baremetal +spec: + # ${System Information/...} are expanded by elemental-register from SMBIOS data, not by render. + machineName: "${CLUSTER_NAME}-${System Information/UUID}" + machineInventoryLabels: + app.kubernetes.io/part-of: cluster-api-provider-baremetal + pool.baremetal.alauda.io/eligible: "true" + elemental.cattle.io/serial-number: "${System Information/Serial Number}" + machineInventoryAnnotations: + elemental.cattle.io/machine-uuid: "${System Information/UUID}" + config: + elemental: + install: + device: /dev/elemental-install-target + eject-cd: true + reboot: true + snapshotter: + type: btrfs + maxSnaps: 4 + registration: + auth: tpm + emulate-tpm: true + emulated-tpm-seed: -1 +--- +apiVersion: elemental.cattle.io/v1beta1 +kind: SeedImage +metadata: + name: ${CLUSTER_NAME}-registration-iso + namespace: cpaas-system + labels: + app.kubernetes.io/part-of: cluster-api-provider-baremetal +spec: + type: iso + baseImage: ${REGISTRY_DOMAIN}/tkestack/baremetal-base-image-iso:${OS_IMAGE_TAG} + registrationRef: + apiVersion: elemental.cattle.io/v1beta1 + kind: MachineRegistration + name: ${CLUSTER_NAME}-registration + namespace: cpaas-system + targetPlatform: linux/amd64 + size: 20Gi + cleanupAfterMinutes: 120 + cloud-config: + stages: + boot: + - name: "Size COS_STATE for reprovisioning" + files: + - path: /etc/elemental/config.d/partitions.yaml + permissions: 0644 + content: | + install: + partitions: + state: + size: 20480 +``` + +A workload `MachineRegistration` never carries the `system-agent-auth-scope: global` annotation: its +hosts register through the platform domain and get the shared identity, which keeps them reporting +after a failover. + +`wl-prod/15-var-cpaas-storage.yaml` is identical to `global-primary/15-var-cpaas-storage.yaml`; copy +it. + +`wl-prod/20-pools.yaml`: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: MachineInventoryPool +metadata: + name: ${CLUSTER_NAME}-control-plane-pool + namespace: cpaas-system +spec: + clusterName: ${CLUSTER_NAME} + machineInventories: + - name: ${BM_CP_INVENTORY_1} + hostname: ${BM_CP_HOSTNAME_1} + - name: ${BM_CP_INVENTORY_2} + hostname: ${BM_CP_HOSTNAME_2} + - name: ${BM_CP_INVENTORY_3} + hostname: ${BM_CP_HOSTNAME_3} +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: MachineInventoryPool +metadata: + name: ${CLUSTER_NAME}-infra-pool + namespace: cpaas-system +spec: + clusterName: ${CLUSTER_NAME} + machineInventories: + - name: ${BM_INFRA_INVENTORY_1} + hostname: ${BM_INFRA_HOSTNAME_1} + - name: ${BM_INFRA_INVENTORY_2} + hostname: ${BM_INFRA_HOSTNAME_2} + - name: ${BM_INFRA_INVENTORY_3} + hostname: ${BM_INFRA_HOSTNAME_3} +``` + +`wl-prod/30-control-plane.yaml`: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: BaremetalCluster +metadata: + name: ${CLUSTER_NAME} + namespace: cpaas-system +spec: + controlPlaneLoadBalancer: + type: Internal + host: "${CONTROL_PLANE_VIP}" + port: 6443 + vrid: ${BM_CONTROL_PLANE_VRID} + networkType: kube-ovn +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: BaremetalMachineTemplate +metadata: + name: ${CLUSTER_NAME}-control-plane-template + namespace: cpaas-system +spec: + template: + spec: + machineInventoryPoolRef: + name: ${CLUSTER_NAME}-control-plane-pool +--- +apiVersion: controlplane.cluster.x-k8s.io/v1beta1 +kind: KubeadmControlPlane +metadata: + name: ${CLUSTER_NAME}-control-plane + namespace: cpaas-system +spec: + replicas: 3 + version: "${K8S_VERSION}" + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + machineTemplate: + nodeDrainTimeout: 5m + nodeVolumeDetachTimeout: 5m + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: BaremetalMachineTemplate + name: ${CLUSTER_NAME}-control-plane-template + kubeadmConfigSpec: + users: + - name: boot + sudo: ALL=(ALL) NOPASSWD:ALL + shell: /bin/bash + sshAuthorizedKeys: + - "${BOOT_SSH_PUBLIC_KEY}" + files: + - path: /etc/kubernetes/admission/psa-config.yaml + owner: "root:root" + permissions: "0644" + content: | + apiVersion: apiserver.config.k8s.io/v1 + kind: AdmissionConfiguration + plugins: + - name: PodSecurity + configuration: + apiVersion: pod-security.admission.config.k8s.io/v1 + kind: PodSecurityConfiguration + defaults: + enforce: "privileged" + enforce-version: "latest" + audit: "baseline" + audit-version: "latest" + warn: "baseline" + warn-version: "latest" + exemptions: + usernames: [] + runtimeClasses: [] + namespaces: + - kube-system + - cpaas-system + - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json + owner: "root:root" + permissions: "0644" + content: | + { + "apiVersion": "kubelet.config.k8s.io/v1beta1", + "kind": "KubeletConfiguration", + "imagePullCredentialsVerificationPolicy": "NeverVerify", + "protectKernelDefaults": true, + "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt", + "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key", + "streamingConnectionIdleTimeout": "5m", + "clientCAFile": "/etc/kubernetes/pki/ca.crt" + } + - path: /etc/kubernetes/audit/policy.yaml + owner: "root:root" + permissions: "0644" + content: | + apiVersion: audit.k8s.io/v1 + kind: Policy + omitStages: + - "RequestReceived" + rules: + - level: None + users: + - system:kube-controller-manager + - system:kube-scheduler + - system:serviceaccount:kube-system:endpoint-controller + verbs: ["get", "update"] + namespaces: ["kube-system"] + resources: + - group: "" + resources: ["endpoints"] + - level: None + nonResourceURLs: + - /healthz* + - /version + - /swagger* + - level: None + resources: + - group: "" + resources: ["events"] + - level: None + verbs: ["get", "list", "watch"] + - level: None + resources: + - group: "coordination.k8s.io" + resources: ["leases"] + - level: None + resources: + - group: "authorization.k8s.io" + resources: ["subjectaccessreviews", "selfsubjectaccessreviews"] + - group: "authentication.k8s.io" + resources: ["tokenreviews"] + - level: Metadata + resources: + - group: "" + resources: ["secrets", "configmaps"] + - level: RequestResponse + resources: + - group: "" + - group: "apps" + - group: "rbac.authorization.k8s.io" + - group: "storage.k8s.io" + - group: "networking.k8s.io" + - level: Metadata + postKubeadmCommands: + - chmod 600 /var/lib/kubelet/config.yaml + clusterConfiguration: + imageRepository: cloud.alauda.io/alauda + dns: + imageTag: "${DNS_IMAGE_TAG}" + etcd: + local: + imageTag: "${ETCD_IMAGE_TAG}" + apiServer: + extraArgs: + audit-log-format: json + audit-log-maxage: "30" + audit-log-maxbackup: "10" + audit-log-maxsize: "200" + profiling: "false" + audit-log-mode: batch + audit-log-path: /etc/kubernetes/audit/audit.log + audit-policy-file: /etc/kubernetes/audit/policy.yaml + tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384" + admission-control-config-file: /etc/kubernetes/admission/psa-config.yaml + tls-min-version: VersionTLS12 + kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt + extraVolumes: + - name: vol-dir-0 + hostPath: /etc/kubernetes + mountPath: /etc/kubernetes + pathType: Directory + controllerManager: + extraArgs: + bind-address: "::" + profiling: "false" + tls-min-version: VersionTLS12 + scheduler: + extraArgs: + bind-address: "::" + tls-min-version: VersionTLS12 + profiling: "false" + initConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + node-labels: "kube-ovn/role=master" + joinConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + node-labels: "kube-ovn/role=master" +--- +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: ${CLUSTER_NAME} + namespace: cpaas-system + labels: + cluster-type: ProviderBaremetal + annotations: + capi.cpaas.io/resource-group-version: infrastructure.cluster.x-k8s.io/v1beta1 + capi.cpaas.io/resource-kind: BaremetalCluster + cpaas.io/kube-ovn-join-cidr: "${KUBE_OVN_JOIN_CIDR}" + cpaas.io/kube-ovn-version: "${KUBE_OVN_VERSION}" + cpaas.io/sentry-deploy-type: Baremetal + cpaas.io/alb-address-type: ClusterAddress # required literal of the Bare Metal profile +spec: + clusterNetwork: + pods: + cidrBlocks: + - ${CLUSTER_CIDR} + services: + cidrBlocks: + - ${SERVICE_CIDR} + controlPlaneRef: + apiVersion: controlplane.cluster.x-k8s.io/v1beta1 + kind: KubeadmControlPlane + name: ${CLUSTER_NAME}-control-plane + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: BaremetalCluster + name: ${CLUSTER_NAME} +``` + +`wl-prod/40-infra-pool.yaml`: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: BaremetalMachineTemplate +metadata: + name: ${CLUSTER_NAME}-infra-template + namespace: cpaas-system +spec: + template: + spec: + machineInventoryPoolRef: + name: ${CLUSTER_NAME}-infra-pool +--- +apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 +kind: KubeadmConfigTemplate +metadata: + name: ${CLUSTER_NAME}-worker-bootstrap + namespace: cpaas-system +spec: + template: + spec: + users: + - name: boot + sudo: ALL=(ALL) NOPASSWD:ALL + shell: /bin/bash + sshAuthorizedKeys: + - "${BOOT_SSH_PUBLIC_KEY}" + files: + - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json + owner: "root:root" + permissions: "0644" + content: | + { + "apiVersion": "kubelet.config.k8s.io/v1beta1", + "kind": "KubeletConfiguration", + "imagePullCredentialsVerificationPolicy": "NeverVerify", + "protectKernelDefaults": true, + "staticPodPath": null, + "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt", + "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key", + "streamingConnectionIdleTimeout": "5m", + "clientCAFile": "/etc/kubernetes/pki/ca.crt" + } + joinConfiguration: + patches: + directory: /etc/kubernetes/patches +--- +apiVersion: cluster.x-k8s.io/v1beta1 +kind: MachineDeployment +metadata: + name: ${CLUSTER_NAME}-infra + namespace: cpaas-system +spec: + clusterName: ${CLUSTER_NAME} + replicas: 3 + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + maxUnavailable: 1 + selector: + matchLabels: {} + template: + metadata: + labels: + cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME} + pool.name: ${CLUSTER_NAME}-infra + node-role.kubernetes.io/infra: "" + node-role.kubernetes.io/worker: "" + spec: + clusterName: ${CLUSTER_NAME} + version: "${K8S_VERSION}" + nodeDrainTimeout: 5m + nodeVolumeDetachTimeout: 5m + taints: + - key: node-role.kubernetes.io/infra + value: reserved + effect: NoSchedule + bootstrap: + configRef: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: ${CLUSTER_NAME}-worker-bootstrap + namespace: cpaas-system + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: BaremetalMachineTemplate + name: ${CLUSTER_NAME}-infra-template + namespace: cpaas-system +``` + +#### Apply order + +Everything runs against `global` with `wl-prod.env` loaded. Steps 1 to 3 reuse the commands of the +`global` installation: replace `"${BOOTSTRAP_KUBECONFIG}"` with `"${GLOBAL_KUBECONFIG}"` and the +`global-primary` file names with the `wl-prod` ones. Apply one file at a time and check it before the +next — never the whole directory. + +1. **Registration ISO.** Render and apply `10-registration.yaml` with the SMBIOS-filtered check, wait + for `SeedImageReady=True` with reason `SeedImageBuildSuccess`, and download the ISO and its checksum + within the cleanup window — the commands of [Build the registration ISO](#build-the-registration-iso) + with `seedimage/${CLUSTER_NAME}-registration-iso`. +2. **Hosts.** [Boot and register the hosts](#boot-and-register-the-hosts). The Live ISO must resolve + `PLATFORM_HOST`, because registration goes to the platform domain. +3. **Inventories.** [Record the inventories](#record-the-inventories) into `wl-prod.env`, then + [Prepare /var/cpaas on every inventory](#prepare-varcpaas-on-every-inventory) with the + `hosts/wl-prod/` files and `wl-prod/15-var-cpaas-storage.yaml`. +4. **Pools, control plane, `infra` pool** — in this order, each after the previous one is healthy: + +```bash +# on global +apply_rendered wl-prod.env wl-prod/20-pools.yaml "${GLOBAL_KUBECONFIG}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io \ + "${CLUSTER_NAME}-control-plane-pool" "${CLUSTER_NAME}-infra-pool" # Ready, Total=3, Available=3 + +apply_rendered wl-prod.env wl-prod/30-control-plane.yaml "${GLOBAL_KUBECONFIG}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get kubeadmcontrolplane "${CLUSTER_NAME}-control-plane" -w + +apply_rendered wl-prod.env wl-prod/40-infra-pool.yaml "${GLOBAL_KUBECONFIG}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get machinedeployment "${CLUSTER_NAME}-infra" -w +``` + +Save the workload kubeconfig as in the common +[Kubeconfig convention](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#kubeconfig-convention) +once the control plane is up. Then add the customer's worker pools as in [Worker pools](#worker-pools): +the main line has no untainted node. + +#### Workload cluster checks + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get cluster.cluster.x-k8s.io "${CLUSTER_NAME}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io \ + -l cluster.x-k8s.io/cluster-name="${CLUSTER_NAME}" \ + -o custom-columns='NAME:.metadata.name,PHASE:.status.phase,READY:.status.ready,INVENTORY:.status.machineInventoryRef.name' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get machineinventories.elemental.cattle.io \ + -o custom-columns='NAME:.metadata.name,PLAN:.status.plan.state,STORAGE:.status.storage.phase,ACTIVE:.status.storage.conditions[?(@.type=="StorageActive")].status' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get baremetalcluster "${CLUSTER_NAME}" \ + -o jsonpath='{.spec.controlPlaneLoadBalancer.type}{" "}{.spec.controlPlaneEndpoint.host}{":"}{.spec.controlPlaneEndpoint.port}'; echo +# on the workload cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes -o wide \ + -L node-role.kubernetes.io/infra,node-role.kubernetes.io/worker +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n kube-system get pods -l app=alive -o wide +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get --raw=/version +``` + +Expect: the `Cluster` `Ready`; three control-plane replicas `Ready`; six `BaremetalMachine` objects +`Running` and `ready`, each on its planned inventory; every used inventory with `PLAN=Applied`, +`STORAGE=Active` and `StorageActive=True`; each pool with +`available + allocated + preparing + reprovisioning + unavailable = total`; all nodes `Ready` with the +planned IPs; the `infra` nodes labeled and tainted; the Alive Pods `Ready` and the API reachable +through the VIP. On each control-plane node, `sudo sysctl net.ipv4.conf.all.arp_accept +net.ipv4.vs.conntrack` prints `1` for both. When something stalls, start from the condition reason on +the `BaremetalMachine` +([Common Failure Modes](https://docs.alauda.io/immutable-infra/1.0/manage-nodes/bare-metal.html#common-failure-modes)): +`PoolMissing`, `PoolExhausted`, `ImageCatalogMiss` (delete the `Machine` after fixing the catalog), or +a `Failed` reprovision plan (plan Secret `failed-output`, the host serial console, Registry +reachability). + +### Variants + +#### Single cluster + +`global` doubles as the workload cluster: one bootstrap node, no standby, no `etcd-sync`, and the +application ingress also lands on the `infra` pool (placement per the common article). Remove the DR +increments: + +```diff +# global-primary.env +-BM_DIRECT_API_SERVER='true' +-BM_SPLIT_AUTH_ENABLED='true' ++BM_DIRECT_API_SERVER='false' ++BM_SPLIT_AUTH_ENABLED='false' + BM_SHARED_AUTH_READ_ONLY='false' +``` + +```diff +# global-primary/10-registration.yaml (MachineRegistration) + labels: + app.kubernetes.io/part-of: cluster-api-provider-baremetal +- annotations: +- baremetal.cluster.io/system-agent-auth-scope: global +``` + +```diff +# global-primary/30-control-plane.yaml + networkType: kube-ovn +- encryptionProviderConfigRef: +- name: global-encryption-provider-config + ... +- - path: /etc/kubernetes/pki/sa.key +- ... # both signing-key file entries +- - path: /etc/kubernetes/pki/sa.pub +- ... + ... + serverCertSANs: +- - "${PRIMARY_CLUSTER_VIP}" +- - "${STANDBY_CLUSTER_VIP}" ++ - "${CONTROL_PLANE_VIP}" + - "${PLATFORM_HOST}" +- - "etcd.kube-system" + ... +- service-account-key-file: /etc/kubernetes/pki/sa.pub +- service-account-signing-key-file: /etc/kubernetes/pki/sa.key +- service-account-issuer: https://kubernetes.default.svc.cluster.local +- api-audiences: https://kubernetes.default.svc.cluster.local + ... +- service-account-private-key-file: /etc/kubernetes/pki/sa.key +``` + +```diff +# global-primary/45-import-extra-resources.yaml +- - resource: "secrets" +- names: +- - "global-encryption-provider-config" +- method: kubectl +``` + +Skip [Generate the shared DR material](#generate-the-shared-dr-material) and the two shared Secrets; +the platform certificate may stay self-signed or be the customer's own. Retire the bootstrap cluster +right after its gate in [Retire the bootstrap cluster](#retire-the-bootstrap-cluster). + +#### Dedicated pools + +For larger workload clusters, replace the `infra` pool with `ingress`, `monitor` and `log` pools. Each +pool gets its own inventory pool, machine template and `MachineDeployment`, its own role label and +taint key, plus the `worker` role label; the bootstrap template is shared. Register the extra hosts, +prepare their `/var/cpaas`, and add `BM_INGRESS_INVENTORY_n` / `BM_INGRESS_HOSTNAME_n` (and the +`MONITOR` and `LOG` equivalents) to the cluster file. The `ingress` pool, as a diff of the `infra` +resources: + +```diff +# wl-prod/20-pools.yaml — one pool per role +- name: ${CLUSTER_NAME}-infra-pool ++ name: ${CLUSTER_NAME}-ingress-pool + ... +- - name: ${BM_INFRA_INVENTORY_1} +- hostname: ${BM_INFRA_HOSTNAME_1} ++ - name: ${BM_INGRESS_INVENTORY_1} ++ hostname: ${BM_INGRESS_HOSTNAME_1} +``` + +```diff +# wl-prod/40-infra-pool.yaml — one template and one MachineDeployment per role +- name: ${CLUSTER_NAME}-infra-template ++ name: ${CLUSTER_NAME}-ingress-template + ... +- name: ${CLUSTER_NAME}-infra-pool ++ name: ${CLUSTER_NAME}-ingress-pool + ... +- name: ${CLUSTER_NAME}-infra ++ name: ${CLUSTER_NAME}-ingress + ... +- pool.name: ${CLUSTER_NAME}-infra +- node-role.kubernetes.io/infra: "" ++ pool.name: ${CLUSTER_NAME}-ingress ++ node-role.kubernetes.io/ingress: "" + node-role.kubernetes.io/worker: "" + ... + taints: +- - key: node-role.kubernetes.io/infra ++ - key: node-role.kubernetes.io/ingress + value: reserved + effect: NoSchedule + ... +- name: ${CLUSTER_NAME}-infra-template ++ name: ${CLUSTER_NAME}-ingress-template +``` + +Distinct taint keys make a misplaced plugin `Pending` immediately instead of landing silently on the +wrong pool. Size each dedicated pool exactly to its replicas, for the same reason as the `infra` pool. + +#### Worker pools + +The main line leaves a workload cluster without an untainted node: the control-plane nodes carry the +kubeadm control-plane taint and the `infra` nodes carry `node-role.kubernetes.io/infra`. Add the +customer's worker pools before handing the cluster over. Each one is built like this generic pool from +[Worker Node Deployment](https://docs.alauda.io/immutable-infra/1.0/manage-nodes/bare-metal.html#worker-node-deployment): +no taint, no platform role label, only the `worker` role label that puts the nodes into the default +worker MachineConfigPool. Register the hosts and prepare their `/var/cpaas` as for the `infra` hosts, +add their entries to the cluster file, and size the pool exactly to its replicas. + +```bash +# wl-prod.env — worker pool entries +BM_WORKER_INVENTORY_1='wl-prod-00000000-0000-0000-0000-000000000031' +BM_WORKER_INVENTORY_2='wl-prod-00000000-0000-0000-0000-000000000032' +BM_WORKER_INVENTORY_3='wl-prod-00000000-0000-0000-0000-000000000033' +BM_WORKER_HOSTNAME_1='wl-prod-worker-1' +BM_WORKER_HOSTNAME_2='wl-prod-worker-2' +BM_WORKER_HOSTNAME_3='wl-prod-worker-3' +``` + +`wl-prod/42-worker-pool.yaml` (it reuses the bootstrap template of `40-infra-pool.yaml`): + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: MachineInventoryPool +metadata: + name: ${CLUSTER_NAME}-worker-pool + namespace: cpaas-system +spec: + clusterName: ${CLUSTER_NAME} + machineInventories: + - name: ${BM_WORKER_INVENTORY_1} + hostname: ${BM_WORKER_HOSTNAME_1} + - name: ${BM_WORKER_INVENTORY_2} + hostname: ${BM_WORKER_HOSTNAME_2} + - name: ${BM_WORKER_INVENTORY_3} + hostname: ${BM_WORKER_HOSTNAME_3} +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: BaremetalMachineTemplate +metadata: + name: ${CLUSTER_NAME}-worker-template + namespace: cpaas-system +spec: + template: + spec: + machineInventoryPoolRef: + name: ${CLUSTER_NAME}-worker-pool +--- +apiVersion: cluster.x-k8s.io/v1beta1 +kind: MachineDeployment +metadata: + name: ${CLUSTER_NAME}-workers + namespace: cpaas-system +spec: + clusterName: ${CLUSTER_NAME} + replicas: 3 + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + maxUnavailable: 1 + selector: + matchLabels: {} + template: + metadata: + labels: + cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME} + pool.name: ${CLUSTER_NAME}-workers + node-role.kubernetes.io/worker: "" + spec: + clusterName: ${CLUSTER_NAME} + version: "${K8S_VERSION}" + nodeDrainTimeout: 5m + nodeVolumeDetachTimeout: 5m + bootstrap: + configRef: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: ${CLUSTER_NAME}-worker-bootstrap + namespace: cpaas-system + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: BaremetalMachineTemplate + name: ${CLUSTER_NAME}-worker-template + namespace: cpaas-system +``` + +```bash +# on global +apply_rendered wl-prod.env wl-prod/42-worker-pool.yaml "${GLOBAL_KUBECONFIG}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get machinedeployment "${CLUSTER_NAME}-workers" -w +# on the workload cluster: the new nodes carry the worker label and no taint +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes -l node-role.kubernetes.io/worker \ + -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key' +``` + +#### External load balancer for the control-plane endpoint + +When the customer provides a Layer-4 load balancer for the API, provision it before applying the +control plane: TCP passthrough on `6443`, the VIP as frontend, every control-plane node IP as a +backend, an HTTPS `/healthz` check expecting `200`, reachable from the bootstrap node or `global` and +from all nodes. The load balancer owner adds and removes backends on every control-plane change +([External LoadBalancer Contract](https://docs.alauda.io/immutable-infra/1.0/how-to/control-plane-endpoint.html#external-loadbalancer-contract)). + +```diff +# 30-control-plane.yaml (BaremetalCluster) + controlPlaneLoadBalancer: +- type: Internal ++ type: External + host: "${CONTROL_PLANE_VIP}" + port: 6443 +- vrid: ${BM_CONTROL_PLANE_VRID} +``` + +`BM_CONTROL_PLANE_VRID` is then unused for that cluster. For `global`, the installer request is +unchanged: `console.host` and `cluster.features.ha.vip` stay the VIP; the load balancer must then also +forward the platform ports of +[LoadBalancer Forwarding Rules](https://docs.alauda.io/container_platform/4.4/install/prepare/prerequisites.html#port-forward) +and, in DR, those of [DR Network Requirements](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#network-requirements), +because the platform and the peer side reach `global` through the same VIP. Check before creation with +`nc -vz 6443`, and after the control plane is up with +`curl -kfsS https://:6443/healthz` per backend. + +### Continue with the common steps + +Continue with the provider-neutral steps, keeping these Bare Metal points in mind: + +- [Install the platform plugins](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#install-the-platform-plugins) + — MetalLB in Layer-2 mode announces the ingress VIP by ARP from the `infra` (or `ingress`) nodes, so + the ingress VIP must be in their Layer-2 segment. Monitoring and logging write under the managed + `/var/cpaas` volume; their storage paths cannot be changed after installation. +- [Configure machines](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#configure-machines) + — every `MachineDeployment` in this article carries the `worker` role label that the default worker + MachineConfigPool selects on; a host added later still has to pass the firmware + [clock gate](#pass-the-clock-gate). +- [Size the clusters](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#size-the-clusters) + — on Bare Metal the `/var/cpaas` size is the data disk you declare; `BM_DATA_DISK_MIN_SIZE` is the + floor Prepare enforces. +- [Accept the delivery](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#accept-the-delivery) + — add the Bare Metal checks below. + +Bare Metal acceptance checks: + +1. **Inventories and pools.** Every used inventory `Ready`, plan `Applied`, storage `Active`; every + pool's `Total` equals its replica count, and nothing is `unavailable`. +2. **Host state.** On one node per pool: `findmnt /var/cpaas` shows the managed volume; + `lsblk -o NAME,SIZE,LABEL | grep COS_STATE` shows about 20G; `timedatectl` reports the clock + synchronized; the node IP equals the address recorded in `spec.observedNetwork`. +3. **Reprovision test and inventory return** — on a workload cluster, before handover. Record the + inventory, IP and `/var/cpaas` filesystem UUID (`findmnt -no UUID /var/cpaas`) of one `infra` node, + then delete its `Machine`: + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get machines.cluster.x-k8s.io \ + -l cluster.x-k8s.io/deployment-name="${CLUSTER_NAME}-infra" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system delete machine +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io -w +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get machineinventory.elemental.cattle.io \ + -o jsonpath='{.metadata.annotations}'; echo +``` + +Pass: the old `BaremetalMachine` goes `Preparing`, and the inventory returns to the pool — +`baremetal.alauda.io/allocation-state` back to `Available`, the `baremetal.alauda.io/owner-*` +annotations cleared, the pool annotation kept; the replacement goes `Reprovisioning` → `Running` on +the **same** inventory, with the same IP, the same `/var/cpaas` filesystem UUID and its data, the +node `Ready` with its labels and taint, and the DaemonSets back on it. +4. **VIP failover drill** (`Internal`): find the control-plane node holding the VIP + (`ip -br addr | grep ` on each node), reboot it, and confirm the API stays reachable through + the VIP while that node is down. +5. **Handoff.** In the DR topology, the gate of + [Verify the handoff on both sides](#verify-the-handoff-on-both-sides) passed on both sides. +6. **Archive without secrets.** The parameter and host files, the rendered manifests and the check + outputs go into the delivery archive; the shared DR files, kubeconfigs, the installer request and + any certificate key never do. + +## Related Information + +- [How to Deliver ACP with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md) — the common article +- [How to Deliver ACP on Huawei DCS with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_on_Huawei_DCS_with_Immutable_Infrastructure.md) +- [How to Deliver ACP on Huawei Cloud Stack with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_on_Huawei_Cloud_Stack_with_Immutable_Infrastructure.md) +- [How to Deliver ACP on VMware vSphere with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_on_VMware_vSphere_with_Immutable_Infrastructure.md) +- [Installing the global Cluster](https://docs.alauda.io/immutable-infra/1.0/global/install.html) +- [Global Cluster Disaster Recovery (Immutable Infrastructure)](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html) +- [Creating Clusters on Bare Metal](https://docs.alauda.io/immutable-infra/1.0/create-cluster/bare-metal.html) +- [Managing Nodes on Bare Metal](https://docs.alauda.io/immutable-infra/1.0/manage-nodes/bare-metal.html) +- [Manage Data Disks on Bare-Metal Hosts](https://docs.alauda.io/immutable-infra/1.0/how-to/manage-bare-metal-storage.html) +- [Select a Fixed System Disk on Multi-Disk Bare-Metal Hosts](https://docs.alauda.io/immutable-infra/1.0/how-to/configure-fixed-install-disk-bare-metal.html) +- [Bare Metal Provider](https://docs.alauda.io/immutable-infra/1.0/overview/providers/bare-metal.html) +- [Plan the Control Plane Endpoint](https://docs.alauda.io/immutable-infra/1.0/how-to/control-plane-endpoint.html) +- [Upgrading Clusters on Bare Metal](https://docs.alauda.io/immutable-infra/1.0/upgrade-cluster/bare-metal.html) +- [Global Cluster Disaster Recovery (ACP)](https://docs.alauda.io/container_platform/4.4/install/global_dr.html) diff --git a/docs/en/solutions/acp/How_to_Deliver_ACP_on_Huawei_Cloud_Stack_with_Immutable_Infrastructure.md b/docs/en/solutions/acp/How_to_Deliver_ACP_on_Huawei_Cloud_Stack_with_Immutable_Infrastructure.md new file mode 100644 index 000000000..0ccf18616 --- /dev/null +++ b/docs/en/solutions/acp/How_to_Deliver_ACP_on_Huawei_Cloud_Stack_with_Immutable_Infrastructure.md @@ -0,0 +1,2277 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 4.4.x +--- + +# How to Deliver ACP on Huawei Cloud Stack with Immutable Infrastructure + +## Purpose + +This article is the Huawei Cloud Stack (HCS) part of the standard delivery of Alauda Container +Platform (ACP) 4.4 on Immutable Infrastructure (Alauda OS nodes). It installs the `global` cluster — +a primary and a standby for disaster recovery (DR) — and creates workload clusters on HCS: complete +manifests with `${VAR}` placeholders, the order in which to apply them, and the check that proves +each step worked. + +Read [How to Deliver ACP with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#how-to-use-these-articles) +first. That common article owns the reference design (topologies, node pools, labels and taints, +disks), the parameter files and helpers, the delivery flow, the platform plugins, machine +configuration, sizing and acceptance. This article covers what is specific to HCS and links back for +the rest. + +Versions this article is written for: + +| Component | Version | Why | +| --- | --- | --- | +| ACP | 4.4.x | The release these articles are pinned to | +| Alauda Container Platform HCS Infrastructure Provider | `v1.0.4` or later | First release that selects the ACP 4.4 component repositories (Kube-OVN chart `v4.4`, CoreDNS tags with the `v4.4` suffix, kube-proxy for Kubernetes 1.35) — see the [release notes](https://docs.alauda.io/immutable-infra/1.0/release-notes/huawei_cloud_stack.html#v1-0-4). Pool-managed persistent disks alone need `v1.0.1` or later | +| Alauda Container Platform Kubeadm Provider | The version delivered with your ACP 4.4 release | Must match the Cluster API control plane resources in the manifests | +| Alauda OS | The `qcow2` image published for your ACP 4.4 patch | HCS uses the `qcow2` format; the image fixes the Kubernetes and component versions | + +Scope: + +- Alauda OS nodes only. For a traditional operating system, use the + [standard installation path](https://docs.alauda.io/container_platform/4.4/install/index.html). +- Every cluster's API endpoint is the Elastic Load Balance (ELB) instance that the HCS provider + creates. No commercial load balancer is needed for platform or cluster VIPs. +- Clusters are created with YAML. Application worker pools follow the customer's machines and + application storage is planned separately; both are out of scope beyond the notes below. + +**Validation status.** The manifests are assembled from the ACP 4.4 product documentation and the +provider versions named above. Validate them in a staging environment before a production delivery. + +## Resolution + +### Huawei Cloud Stack-specific design + +#### Control plane endpoint: the ELB the provider creates + +- For every cluster, the HCS provider creates an ELB for the Kubernetes API server and owns its VIP. + You plan addresses, not a load balancer: the VIP (`vipAddress`) in its subnet (`vipSubnetName`) + and, because the ELB runs with Hybrid Load Balancing so that cluster nodes can reach their own API + server through it, two fixed L4 and two fixed L7 virtual subnet IPs. Every subnet that these + addresses or the nodes use must be listed in `HCSCluster.spec.network.subnets`. +- The endpoint is the VIP — an IP address. Leave `vipDomainName` unset: cluster access by domain is + not supported, because the API server certificate can only be issued by the cluster's own CA. Do + not write `controlPlaneEndpoint` in the manifest; the controller fills it from the ELB. The VIP is + written into the API server certificate and every kubeconfig, so freeze it before installation. +- The provider overview also mentions bandwidth and a public-IP preference as ELB inputs. The + documented create flow sets neither, and this design leaves `publicIpOption` unset: the endpoint + is a private VIP inside the customer network. +- **`global` only:** the provider allocates the additional ELB listeners `443` (web console), + `11443` (registry and console) and `2379` (DR etcd synchronization) only when the infrastructure + cluster is named exactly `global`. Another name installs without error and then breaks registry + pulls, DR synchronization and the web console. Port `80` is not among them — plan platform access + over HTTPS. +- Never disable Hybrid Load Balancing on a provider-created ELB; the nodes reach the API server + through it. + +#### Persistent `/var/cpaas` disks + +- `/var/cpaas` on every node is a pool-managed EVS disk declared in + `HCSMachineConfigPool.spec.configs[].persistentDisks[]`. During a rolling replacement the provider + moves it from the old ECS to the new one, keyed by `(hostname, slot)`, so platform state, + monitoring data and the log collector's working data survive node replacement. +- Kafka, Keeper and ClickHouse keep their data on PersistentVolumes of the StorageClasses in the + customer's storage plan, not on `/var/cpaas` (common article, + [Disks](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#disks)). The HCS product + pages mention an HCS EVS CSI only as external storage and do not document its installation, so agree + on the StorageClass with the customer before the logging stage. +- Never declare `/var/cpaas` in `HCSMachineTemplate.spec.template.spec.dataVolumes[]` — data volumes + are recreated with each ECS. Use them only for `/var/lib/etcd`, `/var/lib/kubelet` and + `/var/lib/containerd`, and never declare the same mount path in both places. +- `slot`, `size`, `type`, `format` and `mountPath` are the disk's identity and cannot be changed + after the provider has accepted the entry. Size `/var/cpaas` before the first apply + ([Size the clusters](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#size-the-clusters)); + growing it later is a migration, not an edit. +- Every pool with persistent disks rolls with `maxSurge: 0`: a surge machine cannot take the + hostname, IP and disk slot of the machine it replaces. The field paths differ — + `KubeadmControlPlane.spec.rolloutStrategy.rollingUpdate.maxSurge` (no `maxUnavailable` exists + there) and `MachineDeployment.spec.strategy.rollingUpdate` (`maxSurge: 0`, `maxUnavailable: 1`). + Copying the `MachineDeployment` block into the `KubeadmControlPlane` leaves the control plane on + the default surge. +- From HCS provider `v1.0.4`, Pod logs (`/var/log/pods`) are placed on the `/var/lib/containerd` + data volume when that volume exists; size it accordingly. + +#### Availability zones and host spread + +- One `HCSMachineTemplate` selects one availability zone. All control plane nodes share one zone, + each worker pool sits in one zone, separate pools can use separate zones, and one pool never + spreads across zones. +- Set `availabilityZone` explicitly with the HCS API `ZoneName`, not a tenant console display name. + When it is omitted, the provider takes the first zone HCS reports as available — an order HCS + controls. +- A persistent disk locks the replacement ECS to the disk's zone. Never change the zone of a template + whose pool already owns disks; the replacement fails with an availability zone conflict. +- `HCSCluster.spec.controlPlaneHA` spreads the three control plane ECS instances across physical + hosts through a provider-owned server group — hosts, not zones. This design enables it with + `soft-anti-affinity` (best effort, never blocks creation). Use `anti-affinity` when strict + separation is required and the zone has the capacity; a hard rule that cannot be met fails ECS + creation. See [Control Plane HA Placement Plan](https://docs.alauda.io/immutable-infra/1.0/infrastructure/huawei-cloud-stack.html#control-plane-ha-placement). +- In a DR pair, place the primary and the standby in different failure domains. + +#### Node identity + +- Static IPs and hostnames come from `HCSMachineConfigPool`, one entry per node. With `maxSurge: 0`, + exactly as many entries as replicas is enough. +- Hostnames use lowercase letters, digits, `-` and `.`. A dotted hostname becomes the node's FQDN and + its first label the POSIX hostname — see + [Hostname behavior on the node](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-cloud-stack.html#hostname-behavior-on-the-node). +- Give the primary and the standby `global` different hostnames, as the + [ACP DR notes](https://docs.alauda.io/container_platform/4.4/install/global_dr.html) recommend: + both clusters carry the same resource names (`global`, `global-*`), and persistent disks are keyed + by `(hostname, slot)`. If both sides share one HCS project, validate in staging that their ELBs + and disks stay apart — the product pages do not say how the provider names them. +- Bootstrap data is `cloud-config`; the HCS provider rejects `ignition`. Never set `providerID` or + `serverId` in a template — the provider assigns them. It also injects the kubelet serving + certificate and, on control plane nodes, an encryption provider file (for DR see + [What changes for DR on HCS](#what-changes-for-dr-on-hcs)). +- The HCS manifests take no SSH key. The `boot` user's key (`BOOT_SSH_PUBLIC_KEY`) reaches the nodes + through the common article's machine configuration. + +#### Application ingress on an HCS VPC + +The common article announces the ingress VIP with MetalLB in Layer 2 mode, which answers ARP for an +address that is not bound to the node's NIC. HCS VPC ports may drop traffic for such addresses +(source and destination checking). The product documentation does not cover MetalLB on an HCS VPC, +so treat it as a staging test item: before you install the ingress plugins, have the HCS +administrator confirm how the ingress VIP is permitted on the `infra` nodes' NICs, and test the VIP +from a client network. If the VPC cannot carry it, use the common article's alternative: a +load balancer in front of the `infra` nodes (on HCS, an ELB that the customer creates for application +traffic) with `IngressNginx.spec.controller.hostNetwork: true`. Never combine it with MetalLB on the +same VIP. + +### Parameters + +The parameter files and the `render`, `load_env` and `apply_rendered` helpers are defined in +[Prepare the parameter files](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#prepare-the-parameter-files). +Run every command in Bash from the delivery directory, after loading the helpers with +`. ./acp-helpers.sh` in each new shell. The common +variables keep their meaning. On HCS, `CONTROL_PLANE_VIP` is the ELB VIP +(`HCSCluster.spec.controlPlaneLoadBalancer.vipAddress`), and `CLUSTER_KUBECONFIG` of a `global` side +is written in Step 5. HCS adds these variables to the same files: + +| Variable | Meaning | Example | Where to get it | +| --- | --- | --- | --- | +| `CORE_PACKAGE` | Core Package archive on the bootstrap node | `/root/packages/acp-core-package.tar` | Downloaded with `violet` for your ACP 4.4 patch | +| `KUBEADM_PROVIDER_PACK` | Kubeadm Provider package on the bootstrap node | `/root/packages/cluster-api-provider-kubeadm.amd64.v1.0.14.tgz` | Customer Portal: Alauda Container Platform Kubeadm Provider | +| `KUBEADM_PROVIDER_VERSION` | Kubeadm Provider chart version | `v1.0.14` | Version in the package file name; confirmed by the registry tag list in Step 2 | +| `HCS_PROVIDER_PACK` | HCS provider package on the bootstrap node | `/root/packages/cluster-api-provider-hcs.amd64.v1.0.4.tgz` | Customer Portal: Alauda Container Platform HCS Infrastructure Provider, `v1.0.4` or later | +| `HCS_PROVIDER_VERSION` | HCS provider chart version | `v1.0.4` | Version in the package file name; confirmed in Step 2 | +| `LOCAL_REGISTRY_ADDRESS` | Bootstrap registry as seen from the bootstrap node (package push) | `127.0.0.1:11443` | Fixed value from the installation guide | +| `BOOTSTRAP_REGISTRY_ADDRESS` | Bootstrap registry as seen from Pods in the bootstrap cluster (`AppRelease` `repoURL`) | `172.18.0.1:11443` | Fixed value from the installation guide | +| `INGRESS_CLASS_NAME` | Platform entrance name that the provider charts expect | `global-alb2` | Fixed value from the installation guide. Keep it; it is not the application ingress | +| `BOOTSTRAP_KUBECONFIG` | Kubeconfig of this side's bootstrap cluster, on that side's bootstrap node | `/root/.kube/config` | The path the installation guide copies the bootstrap kubeconfig to (Step 1 below). Empty in workload files | +| `HOST_IP` | Static IP of this side's bootstrap node | `192.0.2.5` | The bootstrap node; must not change until handoff. Empty in workload files | +| `NODE_REGISTRY_ADDRESS` | Registry that provisioned `global` nodes pull from until the `global` registry takes over | `192.0.2.5:11443` | `HOST_IP` plus `:11443`; `:` only, never `/v2/`. Empty in workload files | +| `PROVIDER_SECRET_NAME` | HCS credential Secret, referenced by `HCSCluster.spec.identityRef.name` | `global-hcs-credential` | Choose; `global-` prefix for `global`; identical on both DR sides | +| `HCS_PROJECT_ID` | Resource Space (project) ID | `0123456789abcdef0123456789abcdef` | HCS **My Settings** > **Resource Spaces** — the ID, not the display name | +| `HCS_REGION` | Region API value used by the provider | `region-example-1` | HCS administrator; tenant administrators cannot read it in the HCS UI | +| `HCS_EXTERNAL_GLOBAL_DOMAIN` | HCS platform domain the provider calls | `hcs.example.com` | HCS administrator | +| `HCS_VPC_NAME` | Existing VPC of the cluster | `vpc-example` | HCS network inventory | +| `HCS_SUBNET_NAME` | Subnet of the nodes, the ELB VIP and the ELB L4/L7 IPs | `subnet-example-global` | HCS network inventory; in `HCS_VPC_NAME` | +| `HCS_SECURITY_GROUP_NAME` | Existing security group of the cluster | `sg-example` | HCS network inventory | +| `HCS_ELB_L4_IP_1` | First L4 virtual subnet IP of the control plane ELB | `192.0.2.31` | Free IP in `HCS_SUBNET_NAME` | +| `HCS_ELB_L4_IP_2` | Second L4 virtual subnet IP | `192.0.2.32` | Free IP in `HCS_SUBNET_NAME` | +| `HCS_ELB_L7_IP_1` | First L7 virtual subnet IP | `192.0.2.33` | Free IP in `HCS_SUBNET_NAME` | +| `HCS_ELB_L7_IP_2` | Second L7 virtual subnet IP | `192.0.2.34` | Free IP in `HCS_SUBNET_NAME` | +| `HCS_CP_HA_POLICY` | Control plane host spread (`HCSCluster.spec.controlPlaneHA.policy`) | `soft-anti-affinity` | `soft-anti-affinity` or `anti-affinity` — see the design above | +| `HCS_IMAGE_NAME` | Alauda OS image name in HCS | `alaudaos-4.4-x86_64-example` | Exact name of the uploaded `qcow2` image | +| `HCS_CP_FLAVOR` | ECS flavor of the control plane nodes | `example.8u32g` | HCS administrator: the `Flavor.Name` API value. Size from the common article | +| `HCS_CP_AZ` | Availability zone of the control plane | `az-example-1` | HCS administrator: the `ZoneName` API value; confirm it is available | +| `HCS_INFRA_FLAVOR` | ECS flavor of the `infra` nodes | `example.16u64g` | As `HCS_CP_FLAVOR` | +| `HCS_INFRA_AZ` | Availability zone of the `infra` pool | `az-example-1` | As `HCS_CP_AZ` | +| `HCS_DISK_TYPE` | EVS disk type for root, data and persistent disks | `SSD` | Disk types available in the target zones | +| `CP_CPAAS_DISK_GB` | Size of `/var/cpaas` on control plane nodes, GB | `100` | Common article sizing; `10`–`32768`; cannot be changed later | +| `INFRA_CPAAS_DISK_GB` | Size of `/var/cpaas` on `infra` nodes, GB | `500` | Common article sizing: platform state, monitoring data and collector working data (plus Kafka and ClickHouse volumes only if their StorageClass provisions on this disk); cannot be changed later | +| `CP1_HOSTNAME` | Hostname of control plane node 1 | `global-primary-cp-1` | Node naming plan; different on each DR side | +| `CP1_IP` | IP of control plane node 1 | `192.0.2.11` | Free IP in `HCS_SUBNET_NAME` | +| `CP2_HOSTNAME` | Hostname of control plane node 2 | `global-primary-cp-2` | Node naming plan | +| `CP2_IP` | IP of control plane node 2 | `192.0.2.12` | Free IP in `HCS_SUBNET_NAME` | +| `CP3_HOSTNAME` | Hostname of control plane node 3 | `global-primary-cp-3` | Node naming plan | +| `CP3_IP` | IP of control plane node 3 | `192.0.2.13` | Free IP in `HCS_SUBNET_NAME` | +| `INFRA1_HOSTNAME` | Hostname of `infra` node 1 | `global-primary-infra-1` | Node naming plan | +| `INFRA1_IP` | IP of `infra` node 1 | `192.0.2.14` | Free IP in `HCS_SUBNET_NAME` | +| `INFRA2_HOSTNAME` | Hostname of `infra` node 2 | `global-primary-infra-2` | Node naming plan | +| `INFRA2_IP` | IP of `infra` node 2 | `192.0.2.15` | Free IP in `HCS_SUBNET_NAME` | +| `INFRA3_HOSTNAME` | Hostname of `infra` node 3 | `global-primary-infra-3` | Node naming plan | +| `INFRA3_IP` | IP of `infra` node 3 | `192.0.2.16` | Free IP in `HCS_SUBNET_NAME` | +| `IMAGE_REPOSITORY` | Repository of the kubeadm-managed images (`clusterConfiguration.imageRepository`) | `cloud.alauda.io/alauda` | Approved release baseline. The DCS and Bare Metal guides use this value, which matches the images preloaded in Alauda OS; confirm it for your release | +| `DNS_IMAGE_TAG` | CoreDNS image tag | `1.14.3-v4.4.6` | [OS Support Matrix](https://docs.alauda.io/immutable-infra/1.0/overview/os-support-matrix.html), row of your ACP 4.4 patch (4.4.0 shown) | +| `ETCD_IMAGE_TAG` | etcd image tag | `v3.6.13-acp.0` | OS Support Matrix, same row | +| `KUBE_OVN_VERSION` | Kube-OVN chart version | `v4.4.6` | OS Support Matrix, same row (`kube-ovn (chart)`) | +| `PRIMARY_CLUSTER_VIP` | ELB VIP of the primary `global` | `192.0.2.10` | Equals `CONTROL_PLANE_VIP` in `global-primary.env` | +| `STANDBY_CLUSTER_VIP` | ELB VIP of the standby `global` | `203.0.113.10` | Equals `CONTROL_PLANE_VIP` in `global-standby.env` | + +Runtime secrets — typed at a prompt or read from a protected file when needed, never stored in a +parameter file (in addition to the common article's `REGISTRY_USERNAME`, `REGISTRY_PASSWORD` and +`ACP_TOKEN`): + +| Variable | Meaning | Example | Where to get it | +| --- | --- | --- | --- | +| `HCS_ACCESS_KEY` | Access key ID of the HCS user the provider acts as | (never stored) | HCS **My Settings** > **Access Keys** | +| `HCS_SECRET_KEY` | Secret access key of that user | (never stored) | As above | +| `PLATFORM_ADMIN_PASSWORD` | Password of the platform administrator `admin@cpaas.io`, set by the installer | (never stored) | Chosen by the customer; the same on both DR sides | +| `PLATFORM_ADMIN_PASSWORD_B64` | Base64 form of it for the installer request | (never stored) | Computed in Step 8 | +| `REGISTRY_PASSWORD_B64` | Base64 form of the platform registry password for the installer request | (never stored) | Computed in Step 8 | +| `ACTIVE_CLUSTER_TOKEN` | Bearer token of the primary's `cpaas-system/k8sadmin` ServiceAccount for etcd-sync | (never stored) | Read from the primary in [Install etcd-sync](#install-etcd-sync-acp-440) | +| `ENCRYPTION_PROVIDER_SECRET_B64` | Shared etcd encryption key (base64 of 32 random bytes) | (never stored) | Generated once in [Prepare the shared DR values](#prepare-the-shared-dr-values); lives only in `/root/dr/encryption-key.b64` | + +Two manifests contain intentional `${...}` that `render` leaves in place: `12-kcp.yaml` of `global` +(`ENCRYPTION_PROVIDER_SECRET_B64`) and `40-installer-config.json` (`REGISTRY_USERNAME`, +`PLATFORM_ADMIN_PASSWORD_B64`, `REGISTRY_PASSWORD_B64`). The steps that use them fill them at apply +time and never write the result to the delivery directory. + +HCS additions to `site.env`, identical on every bootstrap node and delivery host: + +```bash +CORE_PACKAGE='/root/packages/acp-core-package.tar' +KUBEADM_PROVIDER_PACK='/root/packages/cluster-api-provider-kubeadm.amd64.v1.0.14.tgz' +KUBEADM_PROVIDER_VERSION='v1.0.14' +HCS_PROVIDER_PACK='/root/packages/cluster-api-provider-hcs.amd64.v1.0.4.tgz' +HCS_PROVIDER_VERSION='v1.0.4' +LOCAL_REGISTRY_ADDRESS='127.0.0.1:11443' +BOOTSTRAP_REGISTRY_ADDRESS='172.18.0.1:11443' +INGRESS_CLASS_NAME='global-alb2' +HCS_PROJECT_ID='0123456789abcdef0123456789abcdef' +HCS_REGION='region-example-1' +HCS_EXTERNAL_GLOBAL_DOMAIN='hcs.example.com' +HCS_IMAGE_NAME='alaudaos-4.4-x86_64-example' +HCS_DISK_TYPE='SSD' +IMAGE_REPOSITORY='cloud.alauda.io/alauda' +DNS_IMAGE_TAG='1.14.3-v4.4.6' +ETCD_IMAGE_TAG='v3.6.13-acp.0' +KUBE_OVN_VERSION='v4.4.6' +PRIMARY_CLUSTER_VIP='192.0.2.10' +STANDBY_CLUSTER_VIP='203.0.113.10' +``` + +HCS additions to `global-primary.env` (its common part has `CLUSTER_NAME='global'` and +`CONTROL_PLANE_VIP='192.0.2.10'`): + +```bash +BOOTSTRAP_KUBECONFIG='/root/.kube/config' +HOST_IP='192.0.2.5' +NODE_REGISTRY_ADDRESS='192.0.2.5:11443' +PROVIDER_SECRET_NAME='global-hcs-credential' +HCS_VPC_NAME='vpc-example' +HCS_SUBNET_NAME='subnet-example-global' +HCS_SECURITY_GROUP_NAME='sg-example' +HCS_ELB_L4_IP_1='192.0.2.31' +HCS_ELB_L4_IP_2='192.0.2.32' +HCS_ELB_L7_IP_1='192.0.2.33' +HCS_ELB_L7_IP_2='192.0.2.34' +HCS_CP_HA_POLICY='soft-anti-affinity' +HCS_CP_FLAVOR='example.8u32g' +HCS_CP_AZ='az-example-1' +HCS_INFRA_FLAVOR='example.16u64g' +HCS_INFRA_AZ='az-example-1' +CP_CPAAS_DISK_GB='100' +INFRA_CPAAS_DISK_GB='500' +CP1_HOSTNAME='global-primary-cp-1' +CP1_IP='192.0.2.11' +CP2_HOSTNAME='global-primary-cp-2' +CP2_IP='192.0.2.12' +CP3_HOSTNAME='global-primary-cp-3' +CP3_IP='192.0.2.13' +INFRA1_HOSTNAME='global-primary-infra-1' +INFRA1_IP='192.0.2.14' +INFRA2_HOSTNAME='global-primary-infra-2' +INFRA2_IP='192.0.2.15' +INFRA3_HOSTNAME='global-primary-infra-3' +INFRA3_IP='192.0.2.16' +``` + +`global-standby.env` is a copy of `global-primary.env`: keep `CLUSTER_NAME`, +`PROVIDER_SECRET_NAME` and `BOOTSTRAP_KUBECONFIG` (the same path, on the standby's own bootstrap +node), change the common values as the common article describes, and change these HCS values +(network and zone values too if the standby sits elsewhere): + +```bash +HOST_IP='203.0.113.5' +NODE_REGISTRY_ADDRESS='203.0.113.5:11443' +HCS_SUBNET_NAME='subnet-example-standby' +HCS_CP_AZ='az-example-2' +HCS_INFRA_AZ='az-example-2' +HCS_ELB_L4_IP_1='203.0.113.31' +HCS_ELB_L4_IP_2='203.0.113.32' +HCS_ELB_L7_IP_1='203.0.113.33' +HCS_ELB_L7_IP_2='203.0.113.34' +CP1_HOSTNAME='global-standby-cp-1' +CP1_IP='203.0.113.11' +CP2_HOSTNAME='global-standby-cp-2' +CP2_IP='203.0.113.12' +CP3_HOSTNAME='global-standby-cp-3' +CP3_IP='203.0.113.13' +INFRA1_HOSTNAME='global-standby-infra-1' +INFRA1_IP='203.0.113.14' +INFRA2_HOSTNAME='global-standby-infra-2' +INFRA2_IP='203.0.113.15' +INFRA3_HOSTNAME='global-standby-infra-3' +INFRA3_IP='203.0.113.16' +``` + +HCS additions to a workload cluster file, here `wl-prod.env` (its common part has +`CLUSTER_NAME='wl-prod'` and `CONTROL_PLANE_VIP='192.0.2.20'`): + +```bash +BOOTSTRAP_KUBECONFIG='' +HOST_IP='' +NODE_REGISTRY_ADDRESS='' +PROVIDER_SECRET_NAME='global-hcs-credential' +HCS_VPC_NAME='vpc-example' +HCS_SUBNET_NAME='subnet-example-wl-prod' +HCS_SECURITY_GROUP_NAME='sg-example' +HCS_ELB_L4_IP_1='192.0.2.41' +HCS_ELB_L4_IP_2='192.0.2.42' +HCS_ELB_L7_IP_1='192.0.2.43' +HCS_ELB_L7_IP_2='192.0.2.44' +HCS_CP_HA_POLICY='soft-anti-affinity' +HCS_CP_FLAVOR='example.8u32g' +HCS_CP_AZ='az-example-1' +HCS_INFRA_FLAVOR='example.16u64g' +HCS_INFRA_AZ='az-example-1' +CP_CPAAS_DISK_GB='100' +INFRA_CPAAS_DISK_GB='500' +CP1_HOSTNAME='wl-prod-cp-1' +CP1_IP='192.0.2.21' +CP2_HOSTNAME='wl-prod-cp-2' +CP2_IP='192.0.2.22' +CP3_HOSTNAME='wl-prod-cp-3' +CP3_IP='192.0.2.23' +INFRA1_HOSTNAME='wl-prod-infra-1' +INFRA1_IP='192.0.2.24' +INFRA2_HOSTNAME='wl-prod-infra-2' +INFRA2_IP='192.0.2.25' +INFRA3_HOSTNAME='wl-prod-infra-3' +INFRA3_IP='192.0.2.26' +``` + +### Prepare Huawei Cloud Stack + +#### Collect the HCS facts + +Collect every value before you write a file; several can only be read by the HCS administrator. The +full checklist is +[Infrastructure Resources for Huawei Cloud Stack](https://docs.alauda.io/immutable-infra/1.0/infrastructure/huawei-cloud-stack.html). + +| Fact | Goes into | Notes | +| --- | --- | --- | +| HCS user (IAM) and its access key pair | Credential Secret `accessKey`, `secretKey` | **My Settings** > **Access Keys** of the user the provider acts as. The provider creates and deletes ECS instances, EVS disks, the control plane ELB and the control plane server group, and reads the VPC, subnets and security group; the product documentation names no minimum policy, so agree the role with the HCS administrator | +| Project (Resource Space) ID, region API value, HCS platform domain | `HCS_PROJECT_ID`, `HCS_REGION`, `HCS_EXTERNAL_GLOBAL_DOMAIN` | The region value is not visible to tenant administrators | +| HCS API and IAM endpoints | Network paths and DNS | Host names and ports that the provider reaches; the IAM endpoint uses `https` unless the administrator says otherwise | +| VPC, subnets, security group | `HCS_VPC_NAME`, `HCS_SUBNET_NAME`, `HCS_SECURITY_GROUP_NAME` | All subnets of one cluster in one VPC, and mutually reachable | +| Free IPs per cluster | ELB VIP, 2 L4 + 2 L7 ELB IPs, one IP per node | Reserve them in the HCS subnet plan so nothing else takes them | +| Flavors and availability zones | `HCS_*_FLAVOR`, `HCS_*_AZ` | The `Flavor.Name` and `ZoneName` API values, not the tenant console display names | +| EVS disk type | `HCS_DISK_TYPE` | Must exist in every zone you use | + +#### Upload the Alauda OS image + +Download the `qcow2` Alauda OS image for your ACP 4.4 patch from the OS download page of the Customer +Portal ([Downloading Alauda OS](https://docs.alauda.io/immutable-infra/1.0/overview/os-support-matrix.html#downloading-alauda-os)), +upload it to the HCS image service of the project, and record its exact name as `HCS_IMAGE_NAME`. The +image fixes the Kubernetes version, so `K8S_VERSION`, `DNS_IMAGE_TAG` and `ETCD_IMAGE_TAG` must come +from the same OS Support Matrix row. The underlying CPUs must support the `x86-64-v2` baseline. + +#### Open the network paths + +Confirm each path with the customer's network team before installing, in addition to the flows in +the common article's [Reference design](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#reference-design): + +- **Bootstrap node → HCS API and IAM endpoints.** The providers run inside the bootstrap cluster + during the `global` installation. See the bootstrap + [Network](https://docs.alauda.io/immutable-infra/1.0/global/install.html#bootstrap-host-network) + table, which also lists bootstrap node → `global` VIP `6443` and `global` nodes → bootstrap node + `11443`. +- **`global` nodes → HCS API and IAM endpoints.** The product table lists only the bootstrap path, + but after handoff `global` owns its providers + ([Installation Lifecycle](https://docs.alauda.io/immutable-infra/1.0/global/install.html#installation-lifecycle)), + which create workload clusters and replace nodes from there. Test this path from a `global` node + before the first scale-out. +- **`global` nodes → each workload cluster's ELB VIP `6443`.** Cluster API on `global` manages the + workload control plane through it + ([Plan the Control Plane Endpoint](https://docs.alauda.io/immutable-infra/1.0/how-to/control-plane-endpoint.html#external-loadbalancer-contract)). +- **DNS.** The platform domain resolves for users, bootstrap nodes and all cluster nodes; the HCS + endpoint names resolve for the bootstrap nodes and the `global` nodes. Nodes use the DNS servers of + their HCS subnet. +- **Security group.** It must allow the cluster-internal traffic and the ELB's traffic to the control + plane nodes on the listener ports. Use the + [ACP 4.4 Communication Matrix](https://docs.alauda.io/container_platform/4.4/install/prepare/communication_matrix.html) + for port approvals and read its [list of flows it does not cover](https://docs.alauda.io/container_platform/4.4/install/prepare/communication_matrix.html#not_covered) + (traffic between `global` and workload clusters, load balancer forwarding, DR). +- **DR.** Allow `443`, `6443`, `11443` and `2379` in both directions between the primary and standby + networks — see the [DR network requirements](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#network-requirements). + On HCS the ELB listeners themselves come from the provider; only the network rules are yours to + prepare. + +#### Prepare the provider packages + +Download the two cluster plugins — **Alauda Container Platform Kubeadm Provider** and **Alauda +Container Platform HCS Infrastructure Provider** (`v1.0.4` or later) — from the Customer Portal, and +copy them with the Core Package to the paths in `site.env` on each bootstrap node. The bootstrap +cluster installs both providers from the file below. Its values are fixed by the installation guide; +`global.cluster.type: Baremetal` is a chart-internal classifier, not the IaaS name. + +`global-primary/00-provider-appreleases.yaml` + +```yaml +apiVersion: operator.alauda.io/v1alpha1 +kind: AppRelease +metadata: + annotations: + auto-recycle: "true" + interval-sync: "true" + name: cluster-api-provider-kubeadm + namespace: cpaas-system +spec: + destination: + cluster: "" + namespace: "" + source: + charts: + - name: ait/chart-cluster-api-provider-kubeadm + releaseName: cluster-api-provider-kubeadm + targetRevision: ${KUBEADM_PROVIDER_VERSION} + repoURL: ${BOOTSTRAP_REGISTRY_ADDRESS} + timeout: 120 + values: + global: + albName: ${INGRESS_CLASS_NAME} + auth: + default_admin: admin@cpaas.io + cluster: + isGlobal: true + name: global + networkType: kube-ovn + type: Baremetal + host: ${PLATFORM_HOST} + ingress: + ingressClassName: ${INGRESS_CLASS_NAME} + labelBaseDomain: cpaas.io + namespace: cpaas-system + platformUrl: https://${PLATFORM_HOST} + protectSecretFiles: + enabled: false + region: global + registry: + address: ${BOOTSTRAP_REGISTRY_ADDRESS} + replicas: 1 + scheme: https +--- +apiVersion: operator.alauda.io/v1alpha1 +kind: AppRelease +metadata: + annotations: + auto-recycle: "true" + interval-sync: "true" + name: cluster-api-provider-hcs + namespace: cpaas-system +spec: + destination: + cluster: "" + namespace: "" + source: + charts: + - name: ait/chart-cluster-api-provider-hcs + releaseName: cluster-api-provider-hcs + targetRevision: ${HCS_PROVIDER_VERSION} + repoURL: ${BOOTSTRAP_REGISTRY_ADDRESS} + timeout: 120 + values: + global: + albName: ${INGRESS_CLASS_NAME} + auth: + default_admin: admin@cpaas.io + cluster: + isGlobal: true + name: global + networkType: kube-ovn + type: Baremetal + host: ${PLATFORM_HOST} + ingress: + ingressClassName: ${INGRESS_CLASS_NAME} + labelBaseDomain: cpaas.io + namespace: cpaas-system + platformUrl: https://${PLATFORM_HOST} + protectSecretFiles: + enabled: false + region: global + registry: + address: ${BOOTSTRAP_REGISTRY_ADDRESS} + replicas: 1 + scheme: https +``` + +The embedded bootstrap registry is anonymous by default. Only if you set a registry user and password +during `setup.sh`, add `spec.source.chartPullSecret: global-registry-auth` and +`spec.values.global.registry.imagePullSecrets: [global-registry-auth]` to both releases, as Step 3 of +[Installing the global Cluster](https://docs.alauda.io/immutable-infra/1.0/global/install.html) +shows; never create a placeholder Secret. + +#### Create the credential Secret + +The provider reads the HCS credential from a Secret in `cpaas-system` of the cluster it runs in: the +bootstrap cluster during the `global` installation, `global` afterwards. Create it with +`kubectl create`, never as a manifest with `kubectl apply` — client-side apply stores the secret key in +the Secret's own `last-applied-configuration` annotation, and base64 is an encoding, not protection. +The keys are passed through process substitution, so they never appear on a command line. +Set `KUBECONFIG_FOR_SECRET` to the kubeconfig of the target cluster first — `${BOOTSTRAP_KUBECONFIG}` +during the `global` installation, `${GLOBAL_KUBECONFIG}` later; the block refuses to run without it. + +```bash +# on the bootstrap cluster (global installation) or on global (later), cluster file loaded +: "${KUBECONFIG_FOR_SECRET:?set KUBECONFIG_FOR_SECRET to the target cluster first}" +read -rsp 'HCS access key ID: ' HCS_ACCESS_KEY; echo +read -rsp 'HCS secret access key: ' HCS_SECRET_KEY; echo +kubectl --kubeconfig "${KUBECONFIG_FOR_SECRET}" -n cpaas-system create secret generic "${PROVIDER_SECRET_NAME}" \ + --from-file=accessKey=<(printf '%s' "${HCS_ACCESS_KEY}") \ + --from-file=secretKey=<(printf '%s' "${HCS_SECRET_KEY}") \ + --from-literal=projectID="${HCS_PROJECT_ID}" \ + --from-literal=region="${HCS_REGION}" \ + --from-literal=externalGlobalDomain="${HCS_EXTERNAL_GLOBAL_DOMAIN}" +unset HCS_ACCESS_KEY HCS_SECRET_KEY +# lets the web UI list this credential (node pool management on global) +kubectl --kubeconfig "${KUBECONFIG_FOR_SECRET}" -n cpaas-system label secret "${PROVIDER_SECRET_NAME}" \ + capi.cpaas.io/provider=cluster-api-provider-hcs +``` + +Add `--from-literal=schema=http` only if the HCS IAM endpoint uses plain HTTP; the default is `https`. + +### Install the global cluster + +The overall order of the delivery is in the common article's +[Delivery flow](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#delivery-flow). +The steps below follow [Installing the global Cluster](https://docs.alauda.io/immutable-infra/1.0/global/install.html), +Steps 1–9, Huawei Cloud Stack tabs, with the `global` resources assembled into complete files. Read +the [naming convention](https://docs.alauda.io/immutable-infra/1.0/global/install.html#common-prerequisites) +first: `Cluster` and `HCSCluster` are named exactly `global`, every other resource starts with +`global-`. A resource without the prefix installs normally and is then invisible to DR — the standby's +machines are deleted at the first failover. + +The manifests are the multi-cluster main line, with the DR increments already in place (marked in +the files). **Before you start, complete [Prepare the shared DR values](#prepare-the-shared-dr-values) +and point the platform domain at the primary VIP.** For a single cluster without DR, use the +[Single cluster](#single-cluster) variant. The steps are shown for the primary; the standby runs the +same steps with `global-standby.env` and `global-standby/` (see +[Install the primary and the standby](#install-the-primary-and-the-standby)). + +Each side needs its own bootstrap node — 4 cores and 8 GB minimum (x86_64), 300 GB disk, a +traditional Linux operating system and a static IP — see +[Bootstrap Node Requirements](https://docs.alauda.io/immutable-infra/1.0/global/install.html#bootstrap-host-requirements). +The simplest placement is an ECS in the same VPC as the `global` nodes. + +#### Step 1: Create the bootstrap cluster + +```bash +# on the bootstrap node, in the delivery directory +load_env global-primary.env +mkdir -p /root/cpaas-install +tar -xvf "${CORE_PACKAGE}" -C /root/cpaas-install +( cd /root/cpaas-install/installer && bash setup.sh ) +if ! command -v kubectl >/dev/null 2>&1; then + mkdir -p "$HOME/.local/bin" + nerdctl cp minialauda-control-plane:/usr/bin/kubectl "$HOME/.local/bin/kubectl" + chmod 0755 "$HOME/.local/bin/kubectl" + export PATH="$HOME/.local/bin:$PATH" +fi +( umask 077; mkdir -p "$(dirname "${BOOTSTRAP_KUBECONFIG}")"; cp /var/cpaas/data/alauda.kubeconfig "${BOOTSTRAP_KUBECONFIG}" ) +# Steps 4 and 8 call envsubst (gettext) directly +command -v envsubst >/dev/null || echo "envsubst missing - install gettext before Step 4" +# on the bootstrap cluster +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" get nodes +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get pods +``` + +The bootstrap cluster `minialauda` is `Ready` and the installer Pods run. Both `global` cluster +files use the same `BOOTSTRAP_KUBECONFIG` path, so run every `# on the bootstrap cluster` command on +that side's own bootstrap node. + +#### Step 2: Install the provider packages + +```bash +# on the bootstrap node +/root/cpaas-install/installer/res/amd64/packtool pack push \ + -r "${LOCAL_REGISTRY_ADDRESS}" -c "${HCS_PROVIDER_PACK}" +/root/cpaas-install/installer/res/amd64/packtool pack push \ + -r "${LOCAL_REGISTRY_ADDRESS}" -c "${KUBEADM_PROVIDER_PACK}" +# the tags must contain HCS_PROVIDER_VERSION and KUBEADM_PROVIDER_VERSION +curl -s "http://${LOCAL_REGISTRY_ADDRESS}/v2/ait/chart-cluster-api-provider-hcs/tags/list"; echo +curl -s "http://${LOCAL_REGISTRY_ADDRESS}/v2/ait/chart-cluster-api-provider-kubeadm/tags/list"; echo +# on the bootstrap cluster +apply_rendered global-primary.env global-primary/00-provider-appreleases.yaml "${BOOTSTRAP_KUBECONFIG}" +until kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" get crd kubeadmcontrolplanes.controlplane.cluster.x-k8s.io >/dev/null 2>&1; do sleep 10; done +until kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" get crd hcsclusters.infrastructure.cluster.x-k8s.io >/dev/null 2>&1; do sleep 10; done +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get apprelease \ + -o custom-columns='NAME:.metadata.name,REQUESTED:.spec.source.charts[*].targetRevision,INSTALLED:.status.charts[*].installedRevision,PHASE:.status.charts[*].phase' +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" get crd | grep -E '^hcs.*\.infrastructure\.cluster\.x-k8s\.io' +``` + +Both releases report `Success` with the requested revision, and the four `hcs*` CRDs exist +(`hcsclusters`, `hcsmachines`, `hcsmachinetemplates`, `hcsmachineconfigpools`). + +#### Step 3: Create the credential Secret + +Point the credential block at the bootstrap cluster, run +[Create the credential Secret](#create-the-credential-secret), then check the Secret: + +```bash +# on the bootstrap cluster +KUBECONFIG_FOR_SECRET="${BOOTSTRAP_KUBECONFIG}" +# ... the commands of "Create the credential Secret" ... +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get secret "${PROVIDER_SECRET_NAME}" +``` + +#### Step 4: Apply the control plane + +Five files. The pool assigns hostnames, IPs and the `/var/cpaas` disk; the template sets image, +flavor, zone and temporary volumes: + +`global-primary/10-cp-pool.yaml` + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: HCSMachineConfigPool +metadata: + name: global-cp-pool + namespace: cpaas-system + labels: + cluster.x-k8s.io/cluster-name: global +spec: + configs: + - hostname: ${CP1_HOSTNAME} + networks: + - subnetName: ${HCS_SUBNET_NAME} + ipAddress: ${CP1_IP} + persistentDisks: + - slot: 0 + size: ${CP_CPAAS_DISK_GB} + type: ${HCS_DISK_TYPE} + mountPath: /var/cpaas + format: xfs + - hostname: ${CP2_HOSTNAME} + networks: + - subnetName: ${HCS_SUBNET_NAME} + ipAddress: ${CP2_IP} + persistentDisks: + - slot: 0 + size: ${CP_CPAAS_DISK_GB} + type: ${HCS_DISK_TYPE} + mountPath: /var/cpaas + format: xfs + - hostname: ${CP3_HOSTNAME} + networks: + - subnetName: ${HCS_SUBNET_NAME} + ipAddress: ${CP3_IP} + persistentDisks: + - slot: 0 + size: ${CP_CPAAS_DISK_GB} + type: ${HCS_DISK_TYPE} + mountPath: /var/cpaas + format: xfs +``` + +`global-primary/11-cp-template.yaml` + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: HCSMachineTemplate +metadata: + name: global-cp-machine-template + namespace: cpaas-system +spec: + template: + spec: + imageName: ${HCS_IMAGE_NAME} + flavorName: ${HCS_CP_FLAVOR} + availabilityZone: ${HCS_CP_AZ} + rootVolume: + type: ${HCS_DISK_TYPE} + size: 100 + configPoolRef: + name: global-cp-pool + dataVolumes: + - size: 10 + type: ${HCS_DISK_TYPE} + mountPath: /var/lib/etcd + format: xfs + - size: 100 + type: ${HCS_DISK_TYPE} + mountPath: /var/lib/kubelet + format: xfs + - size: 100 + type: ${HCS_DISK_TYPE} + mountPath: /var/lib/containerd + format: xfs +``` + +The `KubeadmControlPlane` body is the one from +[Creating Clusters on Huawei Cloud Stack](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-cloud-stack.html#using-yaml) +with the `global` changes: `global-` names, `format: cloud-config`, the etcd server certificate SANs, +and the encryption provider entry. Keep `apiServer.extraArgs.encryption-provider-config`. The kubelet +setting `imagePullCredentialsVerificationPolicy` is required because ACP 4.4 runs Kubernetes 1.35. + +`global-primary/12-kcp.yaml` + +```yaml +apiVersion: controlplane.cluster.x-k8s.io/v1beta1 +kind: KubeadmControlPlane +metadata: + name: global-kcp + namespace: cpaas-system +spec: + replicas: 3 + version: "${K8S_VERSION}" + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + machineTemplate: + nodeDrainTimeout: 1m + nodeDeletionTimeout: 5m + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: HCSMachineTemplate + name: global-cp-machine-template + kubeadmConfigSpec: + format: cloud-config + files: + - path: /etc/kubernetes/admission/psa-config.yaml + owner: root:root + permissions: "0644" + content: | + apiVersion: apiserver.config.k8s.io/v1 + kind: AdmissionConfiguration + plugins: + - name: PodSecurity + configuration: + apiVersion: pod-security.admission.config.k8s.io/v1 + kind: PodSecurityConfiguration + defaults: + enforce: "privileged" + enforce-version: "latest" + audit: "baseline" + audit-version: "latest" + warn: "baseline" + warn-version: "latest" + exemptions: + usernames: [] + runtimeClasses: [] + namespaces: + - kube-system + - cpaas-system + - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json + owner: root:root + permissions: "0644" + content: | + { + "apiVersion": "kubelet.config.k8s.io/v1beta1", + "kind": "KubeletConfiguration", + "imagePullCredentialsVerificationPolicy": "NeverVerify", + "protectKernelDefaults": true, + "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt", + "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key", + "streamingConnectionIdleTimeout": "5m", + "clientCAFile": "/etc/kubernetes/pki/ca.crt" + } + - path: /etc/kubernetes/audit/policy.yaml + owner: root:root + permissions: "0644" + content: | + apiVersion: audit.k8s.io/v1 + kind: Policy + rules: + - level: Metadata + # DR increment: identical on the primary and the standby. + - path: /etc/kubernetes/encryption-provider.conf + owner: "root:root" + append: false + permissions: "0644" + content: | + apiVersion: apiserver.config.k8s.io/v1 + kind: EncryptionConfiguration + resources: + - resources: + - secrets + providers: + - aescbc: + keys: + - name: key1 + secret: ${ENCRYPTION_PROVIDER_SECRET_B64} + clusterConfiguration: + imageRepository: ${IMAGE_REPOSITORY} + dns: + imageTag: ${DNS_IMAGE_TAG} + etcd: + local: + imageTag: ${ETCD_IMAGE_TAG} + # DR increment: the same list on the primary and the standby. + serverCertSANs: + - "${PRIMARY_CLUSTER_VIP}" + - "${STANDBY_CLUSTER_VIP}" + - "${PLATFORM_HOST}" + - "etcd.kube-system" + apiServer: + extraArgs: + audit-log-format: json + audit-log-mode: batch + audit-log-path: /etc/kubernetes/audit/audit.log + audit-policy-file: /etc/kubernetes/audit/policy.yaml + admission-control-config-file: /etc/kubernetes/admission/psa-config.yaml + encryption-provider-config: /etc/kubernetes/encryption-provider.conf + kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt + profiling: "false" + tls-min-version: VersionTLS12 + extraVolumes: + - name: vol-dir-0 + hostPath: /etc/kubernetes + mountPath: /etc/kubernetes + pathType: Directory + controllerManager: + extraArgs: + bind-address: "::" + flex-volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" + profiling: "false" + tls-min-version: VersionTLS12 + scheduler: + extraArgs: + bind-address: "::" + profiling: "false" + tls-min-version: VersionTLS12 + postKubeadmCommands: + - chmod 600 /var/lib/kubelet/config.yaml + initConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + node-labels: "kube-ovn/role=master" + protect-kernel-defaults: "true" + volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" + joinConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + node-labels: "kube-ovn/role=master" + volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" +``` + +`global-primary/13-hcscluster.yaml` + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: HCSCluster +metadata: + name: global + namespace: cpaas-system +spec: + controlPlaneHA: + enabled: true + policy: ${HCS_CP_HA_POLICY} + controlPlaneLoadBalancer: + vipAddress: ${CONTROL_PLANE_VIP} + vipSubnetName: ${HCS_SUBNET_NAME} + elbVirsubnetL4Ips: + - subnetName: ${HCS_SUBNET_NAME} + ips: + - ${HCS_ELB_L4_IP_1} + - ${HCS_ELB_L4_IP_2} + elbVirsubnetL7Ips: + - subnetName: ${HCS_SUBNET_NAME} + ips: + - ${HCS_ELB_L7_IP_1} + - ${HCS_ELB_L7_IP_2} + networkType: kube-ovn + network: + vpc: + name: ${HCS_VPC_NAME} + subnets: + - name: ${HCS_SUBNET_NAME} + securityGroup: + name: ${HCS_SECURITY_GROUP_NAME} + identityRef: + name: ${PROVIDER_SECRET_NAME} +``` + +If the ELB VIP must sit in another subnet than the nodes, add that subnet as a second +`spec.network.subnets` entry and point `vipSubnetName` (or the L4/L7 `subnetName`) at it — every +referenced subnet must be in the list. + +The `Cluster` carries the HCS annotations of the create-cluster page plus the `global` label and the +temporary registry address (the installer rewrites it to the permanent `global` registry later): + +`global-primary/14-cluster.yaml` + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: global + namespace: cpaas-system + labels: + cluster-type: HCS + is-global: "true" + annotations: + capi.cpaas.io/resource-group-version: infrastructure.cluster.x-k8s.io/v1beta1 + capi.cpaas.io/resource-kind: HCSCluster + cpaas.io/registry-address: "${NODE_REGISTRY_ADDRESS}" + cpaas.io/sentry-deploy-type: Baremetal + cpaas.io/alb-address-type: ClusterAddress + cpaas.io/kube-ovn-join-cidr: ${KUBE_OVN_JOIN_CIDR} + cpaas.io/kube-ovn-version: ${KUBE_OVN_VERSION} +spec: + clusterNetwork: + pods: + cidrBlocks: + - ${CLUSTER_CIDR} + services: + cidrBlocks: + - ${SERVICE_CIDR} + controlPlaneRef: + apiVersion: controlplane.cluster.x-k8s.io/v1beta1 + kind: KubeadmControlPlane + name: global-kcp + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: HCSCluster + name: global +``` + +Check before applying: exactly three control plane entries; no CIDR overlaps the node, VIP or HCS +subnets, another cluster or a customer network; the VIP is an IP address; `/root/dr/encryption-key.b64` +exists. Apply the pool and the template: + +```bash +# on the bootstrap cluster +for f in 10-cp-pool 11-cp-template; do + apply_rendered global-primary.env "global-primary/$f.yaml" "${BOOTSTRAP_KUBECONFIG}" || break +done +``` + +The `KubeadmControlPlane` needs the shared key, which is injected from its protected file into a +rendered copy inside `/root/dr` (mode `700`) and deleted after the apply: + +```bash +# on the bootstrap cluster +if [ -s /root/dr/encryption-key.b64 ]; then + render global-primary.env global-primary/12-kcp.yaml \ + | ENCRYPTION_PROVIDER_SECRET_B64="$(cat /root/dr/encryption-key.b64)" \ + envsubst '${ENCRYPTION_PROVIDER_SECRET_B64}' > /root/dr/12-kcp.yaml + if grep -n '\${' /root/dr/12-kcp.yaml; then + echo "unreplaced variables - nothing applied" + else + kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" apply --dry-run=server -f /root/dr/12-kcp.yaml >/dev/null \ + && kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" apply -f /root/dr/12-kcp.yaml + fi + rm -f /root/dr/12-kcp.yaml +else + echo "shared DR key missing - see Prepare the shared DR values" +fi +``` + +By product design the key becomes part of the `KubeadmControlPlane` object. Treat read access to +`KubeadmControlPlane` objects in `cpaas-system` as access to the key, and archive the template of +`12-kcp.yaml`, never a rendered copy. The common archive scan matches `aescbc` in the `12-kcp.yaml` +templates; every hit must be followed by `${ENCRYPTION_PROVIDER_SECRET_B64}` or +`bootstrap-placeholder`, never a key. When the `KubeadmControlPlane` applied without error, create the +infrastructure cluster and the `Cluster`: + +```bash +# on the bootstrap cluster +for f in 13-hcscluster 14-cluster; do + apply_rendered global-primary.env "global-primary/$f.yaml" "${BOOTSTRAP_KUBECONFIG}" || break +done +``` + +#### Step 5: Wait for the control plane + +```bash +# on the bootstrap cluster +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get cluster global +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get kubeadmcontrolplane global-kcp +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get machines,hcsmachines +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get hcscluster global \ + -o jsonpath='ready={.status.ready} endpoint={.spec.controlPlaneEndpoint.host}:{.spec.controlPlaneEndpoint.port}{"\n"}' +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get hcscluster global \ + -o jsonpath='{range .status.conditions[?(@.type=="ControlPlaneHAReady")]}{.status} {.reason} {.message}{"\n"}{end}' +# on the bootstrap node +curl -sk "https://${CONTROL_PLANE_VIP}:6443/readyz"; echo +``` + +The control plane is ready when `KubeadmControlPlane` is `Ready` with 3 replicas, the `Cluster` is +`Provisioned`, the `HCSCluster` is ready with the endpoint `${CONTROL_PLANE_VIP}:6443`, and +`ControlPlaneHAReady` is `True`. A machine that stays `Pending` shows the HCS reason (quota, network, +credential) on its conditions (`kubectl describe machine`). Save the kubeconfig that Cluster API +publishes — it points at the VIP — as the common article's +[Prepare the parameter files](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#prepare-the-parameter-files) +describes: + +```bash +# on the bootstrap cluster +( umask 077; mkdir -p "$(dirname "${CLUSTER_KUBECONFIG}")" + kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get secret global-kubeconfig \ + -o jsonpath='{.data.value}' | base64 -d > "${CLUSTER_KUBECONFIG}" ) +# on global +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes -o wide +``` + +#### Step 6: Add the infra pool + +Three `infra` nodes carry monitoring and logging of `global`. The role label, the `worker` label (the +default worker MachineConfigPool selects on it) and the taint are declared on the pool: + +`global-primary/20-infra-pool.yaml` + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: HCSMachineConfigPool +metadata: + name: global-infra-pool + namespace: cpaas-system + labels: + cluster.x-k8s.io/cluster-name: global +spec: + configs: + - hostname: ${INFRA1_HOSTNAME} + networks: + - subnetName: ${HCS_SUBNET_NAME} + ipAddress: ${INFRA1_IP} + persistentDisks: + - slot: 0 + size: ${INFRA_CPAAS_DISK_GB} + type: ${HCS_DISK_TYPE} + mountPath: /var/cpaas + format: xfs + - hostname: ${INFRA2_HOSTNAME} + networks: + - subnetName: ${HCS_SUBNET_NAME} + ipAddress: ${INFRA2_IP} + persistentDisks: + - slot: 0 + size: ${INFRA_CPAAS_DISK_GB} + type: ${HCS_DISK_TYPE} + mountPath: /var/cpaas + format: xfs + - hostname: ${INFRA3_HOSTNAME} + networks: + - subnetName: ${HCS_SUBNET_NAME} + ipAddress: ${INFRA3_IP} + persistentDisks: + - slot: 0 + size: ${INFRA_CPAAS_DISK_GB} + type: ${HCS_DISK_TYPE} + mountPath: /var/cpaas + format: xfs +``` + +The temporary volumes use the control plane sizes rather than the 20 GB of the generic worker +example: `infra` nodes pull large platform images, and their Pod logs land on the containerd volume. + +`global-primary/21-infra-template.yaml` + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: HCSMachineTemplate +metadata: + name: global-infra-template + namespace: cpaas-system +spec: + template: + spec: + imageName: ${HCS_IMAGE_NAME} + flavorName: ${HCS_INFRA_FLAVOR} + availabilityZone: ${HCS_INFRA_AZ} + rootVolume: + type: ${HCS_DISK_TYPE} + size: 100 + configPoolRef: + name: global-infra-pool + dataVolumes: + - size: 100 + type: ${HCS_DISK_TYPE} + mountPath: /var/lib/kubelet + format: xfs + - size: 100 + type: ${HCS_DISK_TYPE} + mountPath: /var/lib/containerd + format: xfs +``` + +`global-primary/22-infra-kct.yaml` + +```yaml +apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 +kind: KubeadmConfigTemplate +metadata: + name: global-infra-kct + namespace: cpaas-system +spec: + template: + spec: + files: + - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json + owner: root:root + permissions: "0644" + content: | + { + "apiVersion": "kubelet.config.k8s.io/v1beta1", + "kind": "KubeletConfiguration", + "imagePullCredentialsVerificationPolicy": "NeverVerify", + "protectKernelDefaults": true, + "staticPodPath": null, + "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt", + "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key", + "streamingConnectionIdleTimeout": "5m", + "clientCAFile": "/etc/kubernetes/pki/ca.crt" + } + postKubeadmCommands: + - chmod 600 /var/lib/kubelet/config.yaml + joinConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" +``` + +`global-primary/23-infra-md.yaml` + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: MachineDeployment +metadata: + name: global-infra + namespace: cpaas-system +spec: + clusterName: global + replicas: 3 + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + maxUnavailable: 1 + template: + metadata: + labels: + node-role.kubernetes.io/infra: "" + node-role.kubernetes.io/worker: "" + spec: + clusterName: global + version: "${K8S_VERSION}" + nodeDrainTimeout: 1m + nodeDeletionTimeout: 5m + bootstrap: + configRef: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: global-infra-kct + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: HCSMachineTemplate + name: global-infra-template + taints: + - key: node-role.kubernetes.io/infra + value: reserved + effect: NoSchedule +``` + +```bash +# on the bootstrap cluster +for f in 20-infra-pool 21-infra-template 22-infra-kct 23-infra-md; do + apply_rendered global-primary.env "global-primary/$f.yaml" "${BOOTSTRAP_KUBECONFIG}" || break +done +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get machinedeployment global-infra +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get machines -l cluster.x-k8s.io/cluster-name=global -o wide +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get hcsmachineconfigpool \ + -o custom-columns='NAME:.metadata.name,TOTAL:.status.totalCount,AVAILABLE:.status.availableCount' +# on global +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key' +``` + +Continue when six machines are `Running` (three control plane, three `infra`), `global-infra` reports +3/3 ready, both pools have no available entry left, and the `infra` nodes carry the taint. Their role +labels appear once label synchronization is enabled in the common steps. + +#### Step 7: Import provider resources + +The installer copies the HCS resources and the credential Secret into the new `global` cluster only +if this ConfigMap exists before Step 8 — on every HCS `global` installation, with or without DR. The +`dcs` prefix of its name is historical. A missing ConfigMap is a silent failure: `global` installs, +but its HCS provider has no credential and fails at the first scale-out or node replacement. + +`global-primary/30-import-extra-resources.yaml` + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: dcs-import-extra-resources + namespace: cpaas-system +data: + resources.yaml: | + resources: + - resource: "secrets" + names: ["${PROVIDER_SECRET_NAME}"] + method: kubectl + - resource: "hcsclusters.infrastructure.cluster.x-k8s.io" + names: ["global"] + etcdKeyBase: "/registry/infrastructure.cluster.x-k8s.io/hcsclusters/cpaas-system/" + method: etcdctl + - resource: "hcsmachinetemplates.infrastructure.cluster.x-k8s.io" + etcdKeyBase: "/registry/infrastructure.cluster.x-k8s.io/hcsmachinetemplates/cpaas-system/" + method: etcdctl + - resource: "hcsmachineconfigpools.infrastructure.cluster.x-k8s.io" + etcdKeyBase: "/registry/infrastructure.cluster.x-k8s.io/hcsmachineconfigpools/cpaas-system/" + method: etcdctl + - resource: "hcsmachines.infrastructure.cluster.x-k8s.io" + etcdKeyBase: "/registry/infrastructure.cluster.x-k8s.io/hcsmachines/cpaas-system/" + method: etcdctl +``` + +```bash +# on the bootstrap cluster +apply_rendered global-primary.env global-primary/30-import-extra-resources.yaml "${BOOTSTRAP_KUBECONFIG}" +``` + +#### Step 8: Trigger the platform installation + +The HCS request body has no `cluster` block — the ELB in `HCSCluster` owns the VIP. `console.host` is +the local VIP, never the domain; the platform domain goes into `console.globalHost`. The registry +user and the two passwords are intentional placeholders, filled at submit time: + +`global-primary/40-installer-config.json` + +```json +{ + "basic": { + "username": "admin@cpaas.io", + "password": "${PLATFORM_ADMIN_PASSWORD_B64}" + }, + "registry": { + "domain": "${REGISTRY_DOMAIN}", + "username": "${REGISTRY_USERNAME}", + "password": "${REGISTRY_PASSWORD_B64}" + }, + "console": { + "host": [ + "${CONTROL_PLANE_VIP}" + ], + "globalHost": "${PLATFORM_HOST}", + "httpPort": 80, + "httpsPort": 443, + "cert": { + "selfSigned": {} + } + }, + "product": [ + "base", + "acp" + ], + "deployMode": "normal", + "hostIP": "${HOST_IP}" +} +``` + +In a DR delivery replace `console.cert` with the shared platform certificate first — see +[Prepare the shared DR values](#prepare-the-shared-dr-values). The request is piped straight to the +installer, so the passwords never reach a file: + +```bash +# on the bootstrap node +INSTALLER_IP=$(kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" get pods -n cpaas-system \ + -l service_name=cpaas-installer -o jsonpath='{.items[0].status.podIP}') +# exactly the three intentional placeholders may remain: +render global-primary.env global-primary/40-installer-config.json | grep -n '\${' +read -rsp 'Platform admin password: ' PLATFORM_ADMIN_PASSWORD; echo +read -rp 'Platform registry user: ' REGISTRY_USERNAME +read -rsp 'Platform registry password: ' REGISTRY_PASSWORD; echo +render global-primary.env global-primary/40-installer-config.json \ + | REGISTRY_USERNAME="${REGISTRY_USERNAME}" \ + PLATFORM_ADMIN_PASSWORD_B64="$(printf '%s' "${PLATFORM_ADMIN_PASSWORD}" | base64 -w0)" \ + REGISTRY_PASSWORD_B64="$(printf '%s' "${REGISTRY_PASSWORD}" | base64 -w0)" \ + envsubst '${REGISTRY_USERNAME} ${PLATFORM_ADMIN_PASSWORD_B64} ${REGISTRY_PASSWORD_B64}' \ + | curl -k -X POST "http://${INSTALLER_IP}:8080/cpaas-installer/api/config/dcs" \ + -H 'Content-Type: application/json' -d @- +unset PLATFORM_ADMIN_PASSWORD REGISTRY_USERNAME REGISTRY_PASSWORD +``` + +The endpoint path ends in `/dcs` for every provider. In a DR pair, use the same administrator +password and the same registry user and password on both sides. + +#### Step 9: Monitor and verify + +```bash +# on the bootstrap node +curl -s "http://${INSTALLER_IP}:8080/cpaas-installer/api/progress"; echo +tail -f /var/cpaas/data/installer.log +``` + +A typical installation takes 30–60 minutes; for stalls, use the table under Step 9 of the +installation guide. When the progress API reports `status: Success` and `type: Complete`: + +```bash +# on global +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes -o wide +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get clusters.platform.tkestack.io global -o jsonpath='{.status.phase}{"\n"}' +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get pods --field-selector=status.phase!=Running,status.phase!=Succeeded +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get clustermodule global +# the HCS handoff: credential, HCS resources and providers now live on global +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get secret "${PROVIDER_SECRET_NAME}" +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get hcscluster,hcsmachinetemplate,hcsmachineconfigpool,hcsmachine +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get apprelease \ + -o custom-columns='NAME:.metadata.name,INSTALLED:.status.charts[*].installedRevision,PHASE:.status.charts[*].phase' | grep -E 'NAME|provider' +# on the bootstrap node - the global ELB listeners: any HTTP status proves a listener, a timeout does not +for p in 6443 443 11443; do + curl -sk -o /dev/null -w "$p %{http_code}\n" "https://${CONTROL_PLANE_VIP}:$p/" +done +timeout 3 bash -c " /root/dr/encryption-key.b64 ) +``` + +Copy `/root/dr/encryption-key.b64` to `/root/dr/` (mode `700` directory, mode `600` file) on the +standby bootstrap node over an encrypted channel. It is a secret: never put it into a parameter file, +a ticket or the delivery archive, and keep it only as long as the customer's key-handling policy +allows — rebuilding a side needs it. Then prove that both bootstrap nodes hold the same inputs: + +```bash +# on each bootstrap node, in the delivery directory - the two outputs must be identical +sha256sum /root/dr/encryption-key.b64 site.env +``` + +Platform certificate: the installation guide requires one shared trusted certificate for Bare Metal +DR and is silent for HCS. This design applies the same rule to HCS, because the standby and its +etcd-sync reach the active platform through the one domain. Two unrelated self-signed certificates +would change the certificate behind that domain at every switch. Obtain one trusted certificate whose +SAN covers `${PLATFORM_HOST}` (add a VIP only if clients use HTTPS directly on that VIP). On both sides, +replace `"selfSigned": {}` in `40-installer-config.json` with the `thirdParty` block described under +*Third-Party Console Certificates* in Step 8 of the installation guide. Fill in its certificate and +key fields at submit time, the same way as the passwords. Never store the private key in the +template. + +Lower the TTL of the platform domain's DNS record before you start; the delivery switches it three +times. + +#### Install the primary and the standby + +The order is fixed, and it matches stages 7–12 of the common +[Delivery flow](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#delivery-flow). +What switches is the platform domain; each cluster is always reached by its own VIP. + +1. Point `${PLATFORM_HOST}` at `${PRIMARY_CLUSTER_VIP}` and check it from the primary bootstrap node + with `getent hosts "${PLATFORM_HOST}"`. +2. On the primary bootstrap node, run [Install the global cluster](#install-the-global-cluster) + Steps 1–9 with `global-primary.env` and `global-primary/`. +3. On the primary, complete the common stage 8: upload the packages, enable label synchronization, + install Machine Configuration and configure the machines, then monitoring and logging + ([Install the platform plugins](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#install-the-platform-plugins)). +4. Point `${PLATFORM_HOST}` at `${STANDBY_CLUSTER_VIP}` and check it from the standby bootstrap node. + Several platform resources are rendered with the domain while the standby installer runs, so it + must resolve to the side being installed. +5. On the standby bootstrap node, copy the manifest directory (`cp -r global-primary global-standby`) + and run Steps 1–9 with `global-standby.env` and `global-standby/` — in every command, replace + `global-primary` with `global-standby`. The manifests are identical; the cluster file gives the + standby its own VIP in `console.host` and `vipAddress`, its own `hostIP`, and its own kubeconfigs. + Take `INSTALLER_IP` from the standby bootstrap cluster. +6. On the standby, complete the common stage 10 — the same plugins, uploaded through the standby's + own VIP and registry. +7. Point `${PLATFORM_HOST}` back at `${PRIMARY_CLUSTER_VIP}` **before** installing etcd-sync: the + primary is the source of the first synchronization, and etcd-sync refuses to start when the domain + points at a cluster that runs etcd-sync itself. +8. Prove that both control planes use the same encryption key. Run the command on one control plane + node of each side (log in as `boot`; the key arrived with the machine configuration). The two + hashes must be identical. If they differ, the standby cannot decrypt synchronized Secrets: do not + install etcd-sync. Resolve it with the platform team. + + ```bash + # on a node: one control-plane node of the primary, then one of the standby + sudo sha256sum /etc/kubernetes/encryption-provider.conf + ``` + +9. [Install etcd-sync (ACP 4.4.0+)](#install-etcd-sync-acp-440) on the standby, then + [check the DR pair](#check-the-dr-pair) and retire both bootstrap clusters (Step 10). + +From step 7 on, the commands touch both sides at once. Run them on the delivery host, which holds a +copy of both `global` kubeconfigs (copied from the bootstrap nodes over an encrypted channel), and +name the two kubeconfigs explicitly. Confirm the switch-back from inside both clusters as well, +because cluster DNS can keep the old answer: + +```bash +# on global (both sides), in the delivery directory +load_env global-primary.env +export PRIMARY_GLOBAL_KUBECONFIG=/root/.kube/acp/global-primary.kubeconfig +export STANDBY_GLOBAL_KUBECONFIG=/root/.kube/acp/global-standby.kubeconfig +for kc in "${PRIMARY_GLOBAL_KUBECONFIG}" "${STANDBY_GLOBAL_KUBECONFIG}"; do + kubectl --kubeconfig "$kc" -n cpaas-system exec deployments/sentry -- nslookup "${PLATFORM_HOST}" +done +``` + +Both clusters must resolve `${PLATFORM_HOST}` to `${PRIMARY_CLUSTER_VIP}`. If one does not, restart +its CoreDNS Pods and retry. + +#### Install etcd-sync (ACP 4.4.0+) + +Install the `etcd-sync` plugin on the standby only, as in +[Install etcd-sync (ACP 4.4.0+)](https://docs.alauda.io/immutable-infra/1.0/global/install.html#install-etcd-sync). +The package is published as `etcd-sync` (`packages/etcd-sync//etcd-sync.amd64..tgz`), +not as `global-etcd-sync`, and is not in the installer bundle; the common article's package upload to +the standby (step 6 above) must include it: + +```bash +# on global (standby): the plugin must be published here, not only on the primary +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" get moduleplugins | grep -i etcd +``` + +Create the token Secret on the standby with the primary's `k8sadmin` token. Use `kubectl create`; the +`--dry-run | kubectl apply` form of the product page records the token in +`last-applied-configuration`. The token passes from one cluster to the other through a shell +variable and is never displayed or stored: + +```bash +# on global (primary, then standby) +ACTIVE_CLUSTER_TOKEN=$(kubectl --kubeconfig "${PRIMARY_GLOBAL_KUBECONFIG}" -n cpaas-system \ + get secret k8sadmin -o jsonpath='{.data.token}' | base64 -d) +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system create secret generic \ + etcd-sync-active-cluster-token --from-file=token=<(printf '%s' "${ACTIVE_CLUSTER_TOKEN}") +unset ACTIVE_CLUSTER_TOKEN +``` + +When you repeat this step, delete the old Secret first; `create` does not overwrite. Then, in the +standby web console (`https://${STANDBY_CLUSTER_VIP}`), **Administrator** > **Marketplace** > +**Cluster Plugins**, cluster `global`, install **Alauda Container Platform etcd Synchronizer** with: + +| Parameter | Value | +| --- | --- | +| Active Global Cluster VIP | `${PRIMARY_CLUSTER_VIP}` | +| Active Global Cluster ETCD Endpoints | `https://${PRIMARY_CLUSTER_VIP}:2379` — the primary's HCS ELB forwards `2379` | +| Standby Cluster ETCD Endpoints | Default | +| Active Global Cluster Token Secret | `etcd-sync-active-cluster-token` | +| Data Check Interval | Default | +| Print detail logs | Disabled | + +```bash +# on global (standby) +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system get job etcd-sync-bootstrap +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system get secret remote-etcd-ca remote-etcd-client +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system get issuer remote-etcd-issuer +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system get certificate remote-etcd-client +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system get pods -l app=etcd-sync +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system get lease etcd-sync-mirror -o jsonpath='{.spec.holderIdentity}{"\n"}' +``` + +The monitor Service has only a cluster IP, so run the consistency check on a standby control plane +node. An answer of HTTP `425` means a check is not allowed yet; the loop retries: + +```bash +# on a node: a control-plane node of the standby +mirror_host=$(sudo kubectl -n cpaas-system get svc etcd-sync-monitor -o jsonpath='{.spec.clusterIP}') +case "$mirror_host" in *:*) mirror_host="[$mirror_host]" ;; esac # IPv6 Service address +until code=$(curl -sg -o /tmp/etcd-sync-check.txt -w '%{http_code}' "http://$mirror_host/check") && [ "$code" != 425 ]; do sleep 30; done +cat /tmp/etcd-sync-check.txt +``` + +The bootstrap Job completes, the runtime resources exist, the leader holds the lease, and the check +reports zero missed and zero surplus keys. Missed keys usually clear after restarting the current +leader Pod; surplus keys are resolved with the platform team, never deleted on a guess. Then restart +the Pods that load the DR endpoint configuration, on both sides: + +```bash +# on global (both sides) +for kc in "${PRIMARY_GLOBAL_KUBECONFIG}" "${STANDBY_GLOBAL_KUBECONFIG}"; do + kubectl --kubeconfig "$kc" -n cpaas-system delete po -l 'service_name in (alertmanager,vmselect,vminsert)' + kubectl --kubeconfig "$kc" -n cpaas-system delete po -l service_name=cluster-transformer +done +``` + +#### Check the DR pair + +```bash +# on global (both sides) - each pair of lines must be identical +for kc in "${PRIMARY_GLOBAL_KUBECONFIG}" "${STANDBY_GLOBAL_KUBECONFIG}"; do + kubectl --kubeconfig "$kc" -n cpaas-system get kubeadmcontrolplane global-kcp \ + -o jsonpath='{.spec.kubeadmConfigSpec.clusterConfiguration.etcd.local.serverCertSANs}{"\n"}' +done +for kc in "${PRIMARY_GLOBAL_KUBECONFIG}" "${STANDBY_GLOBAL_KUBECONFIG}"; do + kubectl --kubeconfig "$kc" -n cpaas-system get kubeadmcontrolplane global-kcp \ + -o jsonpath='{.spec.kubeadmConfigSpec.files[?(@.path=="/etc/kubernetes/encryption-provider.conf")].content}' | sha256sum +done +for kc in "${PRIMARY_GLOBAL_KUBECONFIG}" "${STANDBY_GLOBAL_KUBECONFIG}"; do + kubectl --kubeconfig "$kc" -n cpaas-system get secret "${PROVIDER_SECRET_NAME}" -o name +done +# on the delivery host +getent hosts "${PLATFORM_HOST}" +``` + +Pass criteria: both sides passed Step 9; the SAN lists and the declared encryption hashes are +identical, and the node-level check of step 8 passed; the credential Secret exists on both sides; the +platform domain resolves to the primary VIP; the etcd-sync check reports zero differences. The +failover drill is part of acceptance — see +[Continue with the common steps](#continue-with-the-common-steps). Before any failover or DR-aware +upgrade uninstalls etcd-sync, the consistency check must show zero differences: on Immutable +Infrastructure an inconsistent sync can delete workload `Machine` objects and their VMs. + +### Create a workload cluster + +Workload clusters are created on `global` — the active side, the primary in normal operation — from +YAML, as in [Creating Clusters on Huawei Cloud Stack](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-cloud-stack.html#using-yaml). +A guided [web UI](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-cloud-stack.html#using-the-web-ui) +also exists (Fleet Essentials `1.0.2`+ with HCS provider `v1.0.3`+), but its node pool steps do not set +labels or taints, so this article uses YAML. + +#### Before you start + +```bash +# on global +load_env wl-prod.env +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get apprelease \ + -o custom-columns='NAME:.metadata.name,CHART:.spec.source.charts[*].name,INSTALLED:.status.charts[*].installedRevision,PHASE:.status.charts[*].phase' | grep -E 'NAME|provider' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get crd | grep -E '^hcs.*\.infrastructure\.cluster\.x-k8s\.io' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get secret "${PROVIDER_SECRET_NAME}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get deploy capi-controller-manager \ + -o jsonpath='{.spec.template.spec.containers[0].args}{"\n"}' +``` + +The HCS provider is installed at `v1.0.4` or later and `Success`; label synchronization is enabled +(the common stage 8); the credential Secret exists. A workload cluster in the same HCS project and +region references the Secret that `global` already holds. For another project or region, set a new +`PROVIDER_SECRET_NAME` in the cluster file and override `HCS_PROJECT_ID` (and `HCS_REGION`, +`HCS_EXTERNAL_GLOBAL_DOMAIN` if they differ) there. Then run +[Create the credential Secret](#create-the-credential-secret) with +`KUBECONFIG_FOR_SECRET="${GLOBAL_KUBECONFIG}"`, and in a DR pair repeat it against the standby +`global`, with the same name. `cpaas-system` is excluded from etcd synchronization, so a Secret that +exists only on the primary is missing after a failover. The cluster +pulls platform images from the `global` registry by default; see +[Choose the Image Registry for a Workload Cluster](https://docs.alauda.io/immutable-infra/1.0/how-to/workload-cluster-image-registry.html) +for a dedicated registry. + +#### Naming + +`Cluster` and `HCSCluster` are named `${CLUSTER_NAME}` — lowercase letters, digits and hyphens, never +`global` and never starting with `global-`: failover selects resources by that prefix and would treat +them as `global`'s. Every other resource is prefixed `${CLUSTER_NAME}-`. Everything lives in +`cpaas-system`. One manifest set per cluster: copy `wl-prod/` and `wl-prod.env` for the next cluster +and change every value — a value left over from another cluster modifies that cluster's resources. + +#### Manifests + +`wl-prod/10-cp-pool.yaml` + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: HCSMachineConfigPool +metadata: + name: ${CLUSTER_NAME}-cp-pool + namespace: cpaas-system + labels: + cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME} +spec: + configs: + - hostname: ${CP1_HOSTNAME} + networks: + - subnetName: ${HCS_SUBNET_NAME} + ipAddress: ${CP1_IP} + persistentDisks: + - slot: 0 + size: ${CP_CPAAS_DISK_GB} + type: ${HCS_DISK_TYPE} + mountPath: /var/cpaas + format: xfs + - hostname: ${CP2_HOSTNAME} + networks: + - subnetName: ${HCS_SUBNET_NAME} + ipAddress: ${CP2_IP} + persistentDisks: + - slot: 0 + size: ${CP_CPAAS_DISK_GB} + type: ${HCS_DISK_TYPE} + mountPath: /var/cpaas + format: xfs + - hostname: ${CP3_HOSTNAME} + networks: + - subnetName: ${HCS_SUBNET_NAME} + ipAddress: ${CP3_IP} + persistentDisks: + - slot: 0 + size: ${CP_CPAAS_DISK_GB} + type: ${HCS_DISK_TYPE} + mountPath: /var/cpaas + format: xfs +``` + +`wl-prod/11-cp-template.yaml` + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: HCSMachineTemplate +metadata: + name: ${CLUSTER_NAME}-control-plane + namespace: cpaas-system +spec: + template: + spec: + imageName: ${HCS_IMAGE_NAME} + flavorName: ${HCS_CP_FLAVOR} + availabilityZone: ${HCS_CP_AZ} + rootVolume: + type: ${HCS_DISK_TYPE} + size: 100 + configPoolRef: + name: ${CLUSTER_NAME}-cp-pool + dataVolumes: + - size: 10 + type: ${HCS_DISK_TYPE} + mountPath: /var/lib/etcd + format: xfs + - size: 100 + type: ${HCS_DISK_TYPE} + mountPath: /var/lib/kubelet + format: xfs + - size: 100 + type: ${HCS_DISK_TYPE} + mountPath: /var/lib/containerd + format: xfs +``` + +The workload `KubeadmControlPlane` is the create-cluster body with its placeholder encryption entry — +the HCS controller generates the real key for the first control plane machine and reuses it for the +others: + +`wl-prod/12-kcp.yaml` + +```yaml +apiVersion: controlplane.cluster.x-k8s.io/v1beta1 +kind: KubeadmControlPlane +metadata: + name: ${CLUSTER_NAME}-kcp + namespace: cpaas-system +spec: + replicas: 3 + version: "${K8S_VERSION}" + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + machineTemplate: + nodeDrainTimeout: 1m + nodeDeletionTimeout: 5m + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: HCSMachineTemplate + name: ${CLUSTER_NAME}-control-plane + kubeadmConfigSpec: + format: cloud-config + files: + - path: /etc/kubernetes/admission/psa-config.yaml + owner: root:root + permissions: "0644" + content: | + apiVersion: apiserver.config.k8s.io/v1 + kind: AdmissionConfiguration + plugins: + - name: PodSecurity + configuration: + apiVersion: pod-security.admission.config.k8s.io/v1 + kind: PodSecurityConfiguration + defaults: + enforce: "privileged" + enforce-version: "latest" + audit: "baseline" + audit-version: "latest" + warn: "baseline" + warn-version: "latest" + exemptions: + usernames: [] + runtimeClasses: [] + namespaces: + - kube-system + - cpaas-system + - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json + owner: root:root + permissions: "0644" + content: | + { + "apiVersion": "kubelet.config.k8s.io/v1beta1", + "kind": "KubeletConfiguration", + "imagePullCredentialsVerificationPolicy": "NeverVerify", + "protectKernelDefaults": true, + "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt", + "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key", + "streamingConnectionIdleTimeout": "5m", + "clientCAFile": "/etc/kubernetes/pki/ca.crt" + } + # Placeholder only: the HCS controller writes the real file. + - path: /etc/kubernetes/encryption-provider.conf + owner: root:root + permissions: "0600" + content: | + apiVersion: apiserver.config.k8s.io/v1 + kind: EncryptionConfiguration + resources: + - resources: + - secrets + providers: + - aescbc: + keys: + - name: key1 + secret: bootstrap-placeholder + - path: /etc/kubernetes/audit/policy.yaml + owner: root:root + permissions: "0644" + content: | + apiVersion: audit.k8s.io/v1 + kind: Policy + rules: + - level: Metadata + clusterConfiguration: + imageRepository: ${IMAGE_REPOSITORY} + dns: + imageTag: ${DNS_IMAGE_TAG} + etcd: + local: + imageTag: ${ETCD_IMAGE_TAG} + apiServer: + extraArgs: + audit-log-format: json + audit-log-mode: batch + audit-log-path: /etc/kubernetes/audit/audit.log + audit-policy-file: /etc/kubernetes/audit/policy.yaml + admission-control-config-file: /etc/kubernetes/admission/psa-config.yaml + encryption-provider-config: /etc/kubernetes/encryption-provider.conf + kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt + profiling: "false" + tls-min-version: VersionTLS12 + extraVolumes: + - name: vol-dir-0 + hostPath: /etc/kubernetes + mountPath: /etc/kubernetes + pathType: Directory + controllerManager: + extraArgs: + bind-address: "::" + flex-volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" + profiling: "false" + tls-min-version: VersionTLS12 + scheduler: + extraArgs: + bind-address: "::" + profiling: "false" + tls-min-version: VersionTLS12 + postKubeadmCommands: + - chmod 600 /var/lib/kubelet/config.yaml + initConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + node-labels: "kube-ovn/role=master" + protect-kernel-defaults: "true" + volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" + joinConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + node-labels: "kube-ovn/role=master" + volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" +``` + +`wl-prod/13-hcscluster.yaml` + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: HCSCluster +metadata: + name: ${CLUSTER_NAME} + namespace: cpaas-system +spec: + controlPlaneHA: + enabled: true + policy: ${HCS_CP_HA_POLICY} + controlPlaneLoadBalancer: + vipAddress: ${CONTROL_PLANE_VIP} + vipSubnetName: ${HCS_SUBNET_NAME} + elbVirsubnetL4Ips: + - subnetName: ${HCS_SUBNET_NAME} + ips: + - ${HCS_ELB_L4_IP_1} + - ${HCS_ELB_L4_IP_2} + elbVirsubnetL7Ips: + - subnetName: ${HCS_SUBNET_NAME} + ips: + - ${HCS_ELB_L7_IP_1} + - ${HCS_ELB_L7_IP_2} + networkType: kube-ovn + network: + vpc: + name: ${HCS_VPC_NAME} + subnets: + - name: ${HCS_SUBNET_NAME} + securityGroup: + name: ${HCS_SECURITY_GROUP_NAME} + identityRef: + name: ${PROVIDER_SECRET_NAME} +``` + +`wl-prod/14-cluster.yaml` + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: ${CLUSTER_NAME} + namespace: cpaas-system + labels: + cluster-type: HCS + annotations: + cpaas.io/sentry-deploy-type: Baremetal + cpaas.io/alb-address-type: ClusterAddress + capi.cpaas.io/resource-group-version: infrastructure.cluster.x-k8s.io/v1beta1 + capi.cpaas.io/resource-kind: HCSCluster + cpaas.io/kube-ovn-join-cidr: ${KUBE_OVN_JOIN_CIDR} + cpaas.io/kube-ovn-version: ${KUBE_OVN_VERSION} +spec: + clusterNetwork: + pods: + cidrBlocks: + - ${CLUSTER_CIDR} + services: + cidrBlocks: + - ${SERVICE_CIDR} + controlPlaneRef: + apiVersion: controlplane.cluster.x-k8s.io/v1beta1 + kind: KubeadmControlPlane + name: ${CLUSTER_NAME}-kcp + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: HCSCluster + name: ${CLUSTER_NAME} +``` + +The `infra` pool carries ingress, monitoring and logging of the workload cluster: + +`wl-prod/20-infra-pool.yaml` + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: HCSMachineConfigPool +metadata: + name: ${CLUSTER_NAME}-infra-pool + namespace: cpaas-system + labels: + cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME} +spec: + configs: + - hostname: ${INFRA1_HOSTNAME} + networks: + - subnetName: ${HCS_SUBNET_NAME} + ipAddress: ${INFRA1_IP} + persistentDisks: + - slot: 0 + size: ${INFRA_CPAAS_DISK_GB} + type: ${HCS_DISK_TYPE} + mountPath: /var/cpaas + format: xfs + - hostname: ${INFRA2_HOSTNAME} + networks: + - subnetName: ${HCS_SUBNET_NAME} + ipAddress: ${INFRA2_IP} + persistentDisks: + - slot: 0 + size: ${INFRA_CPAAS_DISK_GB} + type: ${HCS_DISK_TYPE} + mountPath: /var/cpaas + format: xfs + - hostname: ${INFRA3_HOSTNAME} + networks: + - subnetName: ${HCS_SUBNET_NAME} + ipAddress: ${INFRA3_IP} + persistentDisks: + - slot: 0 + size: ${INFRA_CPAAS_DISK_GB} + type: ${HCS_DISK_TYPE} + mountPath: /var/cpaas + format: xfs +``` + +`wl-prod/21-infra-template.yaml` + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: HCSMachineTemplate +metadata: + name: ${CLUSTER_NAME}-infra-template + namespace: cpaas-system +spec: + template: + spec: + imageName: ${HCS_IMAGE_NAME} + flavorName: ${HCS_INFRA_FLAVOR} + availabilityZone: ${HCS_INFRA_AZ} + rootVolume: + type: ${HCS_DISK_TYPE} + size: 100 + configPoolRef: + name: ${CLUSTER_NAME}-infra-pool + dataVolumes: + - size: 100 + type: ${HCS_DISK_TYPE} + mountPath: /var/lib/kubelet + format: xfs + - size: 100 + type: ${HCS_DISK_TYPE} + mountPath: /var/lib/containerd + format: xfs +``` + +`wl-prod/22-infra-kct.yaml` + +```yaml +apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 +kind: KubeadmConfigTemplate +metadata: + name: ${CLUSTER_NAME}-infra-kct + namespace: cpaas-system +spec: + template: + spec: + files: + - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json + owner: root:root + permissions: "0644" + content: | + { + "apiVersion": "kubelet.config.k8s.io/v1beta1", + "kind": "KubeletConfiguration", + "imagePullCredentialsVerificationPolicy": "NeverVerify", + "protectKernelDefaults": true, + "staticPodPath": null, + "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt", + "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key", + "streamingConnectionIdleTimeout": "5m", + "clientCAFile": "/etc/kubernetes/pki/ca.crt" + } + postKubeadmCommands: + - chmod 600 /var/lib/kubelet/config.yaml + joinConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" +``` + +`wl-prod/23-infra-md.yaml` + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: MachineDeployment +metadata: + name: ${CLUSTER_NAME}-infra + namespace: cpaas-system +spec: + clusterName: ${CLUSTER_NAME} + replicas: 3 + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + maxUnavailable: 1 + template: + metadata: + labels: + node-role.kubernetes.io/infra: "" + node-role.kubernetes.io/worker: "" + spec: + clusterName: ${CLUSTER_NAME} + version: "${K8S_VERSION}" + nodeDrainTimeout: 1m + nodeDeletionTimeout: 5m + bootstrap: + configRef: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: ${CLUSTER_NAME}-infra-kct + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: HCSMachineTemplate + name: ${CLUSTER_NAME}-infra-template + taints: + - key: node-role.kubernetes.io/infra + value: reserved + effect: NoSchedule +``` + +#### Apply and check + +Apply the control plane first and verify it, then the pool. Never apply the whole directory at once — +pool machines of a broken control plane wait with nothing to join. + +```bash +# on global +for f in 10-cp-pool 11-cp-template 12-kcp 13-hcscluster 14-cluster; do + apply_rendered wl-prod.env "wl-prod/$f.yaml" "${GLOBAL_KUBECONFIG}" || break +done +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get cluster "${CLUSTER_NAME}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get kubeadmcontrolplane "${CLUSTER_NAME}-kcp" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get machines -l cluster.x-k8s.io/cluster-name="${CLUSTER_NAME}" -o wide +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get hcscluster "${CLUSTER_NAME}" \ + -o jsonpath='ready={.status.ready} endpoint={.spec.controlPlaneEndpoint.host}:{.spec.controlPlaneEndpoint.port}{"\n"}' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get hcscluster "${CLUSTER_NAME}" \ + -o jsonpath='{range .status.conditions[?(@.type=="ControlPlaneHAReady")]}{.status} {.reason}{"\n"}{end}' +``` + +When the control plane is `Ready` (3/3) and the endpoint is `${CONTROL_PLANE_VIP}:6443`: + +```bash +# on global +for f in 20-infra-pool 21-infra-template 22-infra-kct 23-infra-md; do + apply_rendered wl-prod.env "wl-prod/$f.yaml" "${GLOBAL_KUBECONFIG}" || break +done +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get machinedeployment "${CLUSTER_NAME}-infra" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get clustermodule "${CLUSTER_NAME}" -o jsonpath='{.status.base.deployStatus}{"\n"}' +( umask 077; kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get secret "${CLUSTER_NAME}-kubeconfig" \ + -o jsonpath='{.data.value}' | base64 -d > "${CLUSTER_KUBECONFIG}" ) +# on the workload cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes -o wide \ + -L node-role.kubernetes.io/infra,node-role.kubernetes.io/worker +``` + +Pass criteria: `Cluster` `Provisioned`/`Running`; six machines `Running`; six nodes `Ready`, the three +`infra` nodes labeled and tainted; `ClusterModule` deploy status `Completed`. If the cluster stays +`Provisioned` with `NotReady` nodes, start with +[Troubleshoot Huawei Cloud Stack Workload Clusters](https://docs.alauda.io/immutable-infra/1.0/how-to/troubleshoot-workload-cluster-huawei-cloud-stack.html). + +The cluster now has only tainted nodes. Add the customer's worker pools before you hand it over — +see [Worker pools](#worker-pools). + +### Variants + +#### Single cluster + +`global` doubles as the workload cluster: one bootstrap node, no standby, no etcd-sync, no workload +cluster creation. Skip [Add disaster recovery](#add-disaster-recovery) and change `12-kcp.yaml`: remove +the `encryption-provider.conf` entry (the HCS controller then generates the key) and use the non-DR +SAN list: + +```yaml +# global-primary/12-kcp.yaml - single cluster: replaces the DR serverCertSANs list + serverCertSANs: + - "${CONTROL_PLANE_VIP}" + - "${PLATFORM_HOST}" +``` + +In Step 4, apply `12-kcp.yaml` with `apply_rendered` like the other files; there is no key to inject. +The `infra` pool also carries ingress +([Install the platform plugins](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#install-the-platform-plugins)). +Its application worker pools ([Worker pools](#worker-pools)) belong to `global` and therefore carry +the `global-` prefix: + +```yaml +# a worker pool on a single-cluster global - differs from 23-infra-md.yaml in these fields +metadata: + name: global-worker +spec: + template: + metadata: + labels: + node-role.kubernetes.io/worker: "" + spec: + bootstrap: + configRef: + name: global-worker-kct + infrastructureRef: + name: global-worker-template + # no taints +``` + +#### Dedicated pools + +For a larger workload cluster, replace the `infra` pool with `ingress`, `monitor` and `log` pools. Copy +`20`–`23` once per pool, replace `infra` with the pool name in every resource name, the role label and +the taint key, give each pool its own node variables (for example `LOG1_HOSTNAME` and `LOG1_IP`) and, +if needed, its own flavor and zone. Each pool keeps `node-role.kubernetes.io/worker` and its own taint +key. The `log` pool's `MachineDeployment`, for example: + +```yaml +# wl-prod/23-log-md.yaml - differs from 23-infra-md.yaml in these fields +metadata: + name: ${CLUSTER_NAME}-log +spec: + template: + metadata: + labels: + node-role.kubernetes.io/log: "" + node-role.kubernetes.io/worker: "" + spec: + bootstrap: + configRef: + name: ${CLUSTER_NAME}-log-kct + infrastructureRef: + name: ${CLUSTER_NAME}-log-template + taints: + - key: node-role.kubernetes.io/log + value: reserved + effect: NoSchedule +``` + +The `HCSMachineTemplate` of each pool references that pool's `HCSMachineConfigPool` +(`configPoolRef.name: ${CLUSTER_NAME}-log-pool`). Number the extra files below `50` (for example +`24`–`35`), because the common article uses `50`–`79` in the same directory. Size each pool's +`/var/cpaas` for what it hosts. The `global` cluster keeps the single `infra` pool in this design. + +#### Worker pools + +The main line creates no untainted node: the control plane nodes carry the kubeadm control plane +taint and the `infra` nodes carry `node-role.kubernetes.io/infra=reserved:NoSchedule`. Before you hand +a workload cluster (or a single-cluster `global`) over, add at least one untainted worker pool sized +to the customer's machines. Build it with the four resources of +[Managing Nodes on Huawei Cloud Stack](https://docs.alauda.io/immutable-infra/1.0/manage-nodes/huawei-cloud-stack.html#using-yaml): +copy `20`–`23` to the next free numbers below `50` (for example `24`–`27`), and replace `infra` with +`worker` in every resource name and in +`configPoolRef.name`. Give the pool one `HCSMachineConfigPool` entry per replica with its own node +variables (for example `WORKER1_HOSTNAME` and `WORKER1_IP`), and its own flavor and zone if needed. +Set `replicas` to the number of entries. Keep `maxSurge: 0`, because the `/var/cpaas` persistent +disk stays on every worker. The `MachineDeployment` drops the role label and the taint, and keeps +`node-role.kubernetes.io/worker` so that the nodes receive machine configuration: + +```yaml +# wl-prod/27-worker-md.yaml - differs from 23-infra-md.yaml in these fields +metadata: + name: ${CLUSTER_NAME}-worker +spec: + template: + metadata: + labels: + node-role.kubernetes.io/worker: "" + spec: + bootstrap: + configRef: + name: ${CLUSTER_NAME}-worker-kct + infrastructureRef: + name: ${CLUSTER_NAME}-worker-template + # no taints +``` + +Check that the new nodes are `Ready` and carry no taint: +`kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key'`. + +### Continue with the common steps + +With the clusters up, continue in the common article on every cluster: + +1. [Install the platform plugins](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#install-the-platform-plugins) + — on HCS, settle the ingress VIP path from + [Application ingress on an HCS VPC](#application-ingress-on-an-hcs-vpc) before MetalLB. +2. [Configure machines](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#configure-machines) + — this also delivers the `boot` user's SSH key used by the node checks below. +3. [Accept the delivery](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#accept-the-delivery), + including the DR failover drill in [Failover](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#failover-procedure). + +HCS-specific acceptance checks, run for each cluster file (`load_env .env`): + +```bash +# on global - every HCS cluster: ready, endpoint = VIP, host spread active +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get hcscluster \ + -o custom-columns='NAME:.metadata.name,READY:.status.ready,ENDPOINT:.spec.controlPlaneEndpoint.host,HA-POLICY:.status.controlPlaneHA.policy' +# every machine: ECS active, planned zone, provider ID set +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get hcsmachine \ + -o custom-columns='NAME:.metadata.name,STATE:.status.instanceState,READY:.status.ready,ZONE:.spec.availabilityZone,PROVIDERID:.spec.providerID' +# every persistent disk: one slot-0 disk per node, Attached, in the planned zone +for p in $(kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get hcsmachineconfigpool -o name); do + echo "== $p" + kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get "$p" \ + -o jsonpath='{range .status.persistentDiskStatus[*]}{.hostname}{"\t"}{.slot}{"\t"}{.phase}{"\t"}{.availabilityZone}{"\n"}{end}' +done +``` + +```bash +# on global - label synchronization is still configured (the provider AppRelease may reconcile the Deployment) +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get deploy capi-controller-manager \ + -o jsonpath='{.spec.template.spec.containers[0].args}{"\n"}' | grep -o -- '--additional-sync-machine-labels=[^"]*' +# on a node (one per pool, as the boot user) - /var/cpaas is the persistent EVS disk, mounted once +findmnt -no SOURCE,FSTYPE,SIZE /var/cpaas +``` + +Pass criteria: every `HCSCluster` ready with its VIP as endpoint; every `HCSMachine` `ACTIVE` in its +planned zone with a provider ID; every node's `/var/cpaas` an attached slot-0 EVS disk of the planned +size, and no other `/var/cpaas` mount; the label synchronization argument still present on each +`global` side. In a DR pair, the node-level encryption check of +[Install the primary and the standby](#install-the-primary-and-the-standby) (step 8) has passed. Rerun +it after any control plane rollout on either side, because the provider rebuilds that file for every +new control plane machine. + +## Related Information + +- [How to Deliver ACP with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md) — the common article +- [How to Deliver ACP on Huawei DCS with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_on_Huawei_DCS_with_Immutable_Infrastructure.md) +- [How to Deliver ACP on Bare Metal with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_on_Bare_Metal_with_Immutable_Infrastructure.md) +- [How to Deliver ACP on VMware vSphere with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_on_VMware_vSphere_with_Immutable_Infrastructure.md) +- [Installing the global Cluster](https://docs.alauda.io/immutable-infra/1.0/global/install.html) +- [Global Cluster Disaster Recovery (Immutable Infrastructure)](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html) +- [Global Cluster Disaster Recovery (ACP, traditional OS)](https://docs.alauda.io/container_platform/4.4/install/global_dr.html#etcd_sync) — synchronized scope and switchover detail +- [Huawei Cloud Stack Provider](https://docs.alauda.io/immutable-infra/1.0/overview/providers/huawei-cloud-stack.html) +- [Huawei Cloud Stack Provider Installation](https://docs.alauda.io/immutable-infra/1.0/install/huawei-cloud-stack.html) +- [Infrastructure Resources for Huawei Cloud Stack](https://docs.alauda.io/immutable-infra/1.0/infrastructure/huawei-cloud-stack.html) +- [Creating Clusters on Huawei Cloud Stack](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-cloud-stack.html) +- [Managing Nodes on Huawei Cloud Stack](https://docs.alauda.io/immutable-infra/1.0/manage-nodes/huawei-cloud-stack.html) +- [Plan the Control Plane Endpoint](https://docs.alauda.io/immutable-infra/1.0/how-to/control-plane-endpoint.html) +- [Configure Annotations, Labels, and Taints on Nodes](https://docs.alauda.io/immutable-infra/1.0/how-to/node-configuration.html) +- [Huawei Cloud Stack Provider Release Notes](https://docs.alauda.io/immutable-infra/1.0/release-notes/huawei_cloud_stack.html) diff --git a/docs/en/solutions/acp/How_to_Deliver_ACP_on_Huawei_DCS_with_Immutable_Infrastructure.md b/docs/en/solutions/acp/How_to_Deliver_ACP_on_Huawei_DCS_with_Immutable_Infrastructure.md new file mode 100644 index 000000000..0c8bdb0b9 --- /dev/null +++ b/docs/en/solutions/acp/How_to_Deliver_ACP_on_Huawei_DCS_with_Immutable_Infrastructure.md @@ -0,0 +1,2081 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 4.4.x +--- + +# How to Deliver ACP on Huawei DCS with Immutable Infrastructure + +## Purpose + +This article is the Huawei DCS part of the standard delivery of Alauda Container Platform (ACP) 4.4 on Immutable Infrastructure. It takes a prepared Huawei Datacenter Virtualization Solution (DCS) site to a `global` cluster pair — primary and standby, with disaster recovery — and to workload clusters managed by `global`. Every cluster gets a three-node control plane and one `infra` pool, built from complete manifests in which you only fill in parameters, and every step ends with the check that proves it worked. + +Read [How to Deliver ACP with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md) first. It defines the reference design, the parameter files and the `render` helper used below, the platform plugins, machine configuration, sizing, and acceptance. This article adds only what is specific to Huawei DCS. + +Versions this article is written for: + +- ACP 4.4, with every node built from the Alauda OS image published for the same ACP release. +- Alauda Container Platform DCS Infrastructure Provider `v1.0.22` or later. `v1.0.21` is the minimum for ACP 4.4 images; `v1.0.22` adds the Self-built VIP that every cluster endpoint in this article uses. +- The Alauda Container Platform Kubeadm Provider delivered with the release. + +Validation status: the manifests are assembled from the ACP 4.4 product documentation and the documentation of DCS Provider `v1.0.22`. Validate them in a staging environment before a production delivery. + +Out of scope: nodes that run a traditional operating system (use the [standard installation path](https://docs.alauda.io/container_platform/4.4/install/index.html)), the sizing of worker pools for applications (they follow the customer's machines; add them before handover, as [Worker pools](#worker-pools) shows), and storage selection. + +## Resolution + +### DCS-specific design + +- **Cluster endpoint: Self-built VIP.** Every cluster sets `DCSCluster.spec.controlPlaneLoadBalancer.type: internal`. The provider brings the VIP up on the first control-plane node during bootstrap, then installs the `alive` runtime (keepalived and IPVS) on every control-plane node through a provider-managed `alive` ModuleInfo, so no load balancer is needed. The VIP must be an unused IPv4 address in the same Layer 2 network as the control-plane NICs, outside every IP pool, with a VRID that is unique in that network, and the DCS port group must allow VIP ownership, GARP/ARP updates, and VRRP. `type`, `host`, `port`, `vrid`, and `interface` cannot change after the control plane is initialized, because the VIP is written into the API server certificate and every kubeconfig. See "Self-built VIP behavior and limitations" under [Configure DCSCluster](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-dcs.html#configure-dcscluster). +- **Alternative endpoint: an existing external load balancer.** If the customer fronts the API with their own load balancer, replace the load-balancer block of `13-dcscluster.yaml` and validate the device against the [external LoadBalancer contract](https://docs.alauda.io/immutable-infra/1.0/how-to/control-plane-endpoint.html#external-loadbalancer-contract) before you apply anything. For `global`, the device must also forward the ports in [LoadBalancer Forwarding Rules](https://docs.alauda.io/container_platform/4.4/install/prepare/prerequisites.html#port-forward), plus `2379` for the DR pair: + + ```yaml + # 13-dcscluster.yaml: external load balancer instead of Self-built VIP + spec: + controlPlaneLoadBalancer: + host: "${CONTROL_PLANE_VIP}" + port: 6443 + type: external + ``` + + Either way the endpoint is an IP address, never a domain name: the API server certificate is issued by the cluster's own CA, so a domain gains nothing and is not supported. +- **Control-plane spread across hosts: a DRS rule.** The provider cannot pin VMs to DCS hosts, so three control-plane VMs can land on one host. `DCSCluster.spec.controlPlaneHA.enabled: true` makes the provider maintain a DRS mutual-exclusion rule for them. It needs DRS enabled on the DCS compute cluster, at least three usable hosts, datastores visible from all of them, and no blocking device such as a mounted CD-ROM. It is independent of the Self-built VIP. See [Cross-Host High Availability for Control Plane](https://docs.alauda.io/immutable-infra/1.0/infrastructure/huawei-dcs.html#cross-host-ha). +- **`/var/cpaas` is a pool-managed persistent disk.** It is declared in `DCSIpHostnamePool.spec.pool[].persistentDisk`, bound to the `(ip, slot)` identity, and reattached to the replacement VM when a node is replaced. Template disks (`/var/lib/etcd`, `/var/lib/kubelet`, `/var/lib/containerd`) are recreated with the VM. Two consequences: + - Every pool rolls with `maxSurge: 0`, because a surge machine cannot take over the IP slot and the disk of the machine it replaces. The field paths differ: `KubeadmControlPlane.spec.rolloutStrategy.rollingUpdate.maxSurge` (no `maxUnavailable` there) and `MachineDeployment.spec.strategy.rollingUpdate.maxSurge` with `maxUnavailable: 1`. Never copy the `MachineDeployment` block into the `KubeadmControlPlane`: it has no `spec.strategy`, and the control plane would silently fall back to surging. + - Entries can be appended but never removed, and `format`, `options`, and `pciType` are fixed at creation, so get the disk right before the first apply. The VM template must be `4.2.1` or later, because safe shutdown and disk detach depend on its guest tools. +- **Datastore binding.** `datastoreName` pins a disk to one datastore, and the DCS scheduler never looks at its siblings; `datastoreClusterName` lets DCS pick a member with free capacity, but every candidate host must see every member. The manifests use `datastoreName`. When the DCS administrator provides a datastore cluster that all candidate hosts can see, replace the key with `datastoreClusterName` in every disk entry of every file. See [Choosing a datastore](https://docs.alauda.io/immutable-infra/1.0/infrastructure/huawei-dcs.html#choosing-datastore). +- **Bootstrap data and encryption at rest.** DCS accepts only `format: ignition`, and the Ignition validator rejects an empty SSH key list, so the `boot` user always carries `BOOT_SSH_PUBLIC_KEY`. The provider writes `/etc/kubernetes/encryption-provider.conf` on control-plane nodes itself and drops any `files` entry for that path; for the DR pair the key is pinned with `DCSCluster.spec.encryptionProviderConfigRef`. See [How the Encryption Provider Configuration Is Delivered](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-dcs.html#encryption-provider-delivery). +- **Provider literals.** `PROVIDER_ID` and `NODE_IP` in the kubelet arguments are replaced by the provider for each machine. They are not `${...}` variables, so `render` leaves them alone; never replace or quote them by hand, or the nodes never register. See [Magic-Token Placeholders](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-dcs.html#magic-tokens). +- **Naming is a hard requirement.** For `global`, the `Cluster` and the `DCSCluster` are named exactly `global` and every other resource starts with `global-`: DR identifies the pair's resources by that prefix, and a resource without it is invisible to DR and loses its machines at the first failover ([naming convention](https://docs.alauda.io/immutable-infra/1.0/global/install.html#common-prerequisites)). A workload cluster is never named `global` and never uses the `global-` prefix ([Workload Cluster Naming](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-dcs.html#workload-cluster-naming)). The `global` manifests below hard-code these names and the workload manifests derive every name from `CLUSTER_NAME`, so a copied file cannot drift. +- **Credentials stay out of manifests.** The DCS API credential Secret is created with `kubectl create`; a Secret applied with `kubectl apply` keeps its content in the `last-applied-configuration` annotation. The installer copies the Secret into `global` through the `dcs-import-extra-resources` ConfigMap, and workload clusters on the same DCS reference that copy instead of writing the password again. +- **One manifest set per cluster.** `global-primary/` and `global-standby/` are byte-identical: every difference between the two sides lives in the cluster env file, and the DR increments read only `site.env`, which is how they stay identical on both sides. A new workload cluster is a copy of `wl-prod/` with its own env file; nothing in the manifests names another cluster. + +### Parameters + +Use the delivery directory, the parameter files, and the `render`, `load_env`, and `apply_rendered` helpers exactly as defined in [Prepare the parameter files](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#prepare-the-parameter-files), and the kubeconfig variables of its [Kubeconfig convention](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#kubeconfig-convention). Copy the delivery directory to each bootstrap node and keep it on the delivery host; run every command from it, after `load_env` of the cluster the command is for. Each command block states where it runs: `# on the bootstrap node` (a shell on that host), `# on the bootstrap cluster` (on that side's bootstrap node, with `BOOTSTRAP_KUBECONFIG`), `# on global`, `# on the workload cluster`, or `# on a node`. + +This article adds these files to each cluster directory, all numbered below `50` (the common article owns `50` to `79`): + +```text +global-primary/ 00 provider AppReleases, 10-14 control plane, 20-23 infra pool, + 30 import ConfigMap, 40 installer request +global-standby/ byte-identical copy: cp -r global-primary global-standby +wl-prod/ 10-14 control plane, 20-23 infra pool + (dedicated layout: 30-33 ingress, 34-37 monitor, 40-43 log; worker pool 44-47) +rendered/ render output (holds no credentials) +``` + +DCS notes on the common variables, which are defined in the [common variables](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#common-variables) tables: + +- `PLATFORM_HOST` is a DNS name. It is the only name that moves during DR. +- `REGISTRY_DOMAIN` is `PLATFORM_HOST` with port `11443`, so both sides of the DR pair register the same platform registry address. +- `CONTROL_PLANE_VIP` is the Self-built VIP of the cluster. In `global-primary.env` set it to `"${PRIMARY_CLUSTER_VIP}"`, in `global-standby.env` to `"${STANDBY_CLUSTER_VIP}"`. +- `CLUSTER_NAME` is `global` in both `global` files. +- `K8S_VERSION` must equal `kubernetesVersion` of the VM template; see [Resolving Placeholder Values](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-dcs.html#resolving-placeholders). +- `BOOT_SSH_PUBLIC_KEY` is mandatory on DCS (see above), and the same key that the common article's machine configuration manages. +- `GLOBAL_KUBECONFIG` and `CLUSTER_KUBECONFIG` both point at the side itself in the two `global` files. For a workload cluster, `GLOBAL_KUBECONFIG` points at the active `global`. + +`site.env` — DCS lines: + +| Variable | Meaning | Example | Where to get it | +| --- | --- | --- | --- | +| `IMAGE_REPOSITORY` | Repository of the kubeadm-managed images (`clusterConfiguration.imageRepository`) | `cloud.alauda.io/alauda` | The value in the product's [Complete KubeadmControlPlane Configuration](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-dcs.html#complete-kubeadmcontrolplane-configuration); keep it unless the release baseline says otherwise | +| `DNS_IMAGE_TAG` | CoreDNS image tag in every `KubeadmControlPlane` | `1.14.3-v4.4.6` | `coredns` column of the [OS Support Matrix](https://docs.alauda.io/immutable-infra/1.0/overview/os-support-matrix.html) row of your ACP release, or `corednsTag` in the `cpaas.io/dcs-vm-template` ConfigMap | +| `ETCD_IMAGE_TAG` | etcd image tag in every `KubeadmControlPlane` | `v3.6.13-acp.0` | `etcd` column of the same row, or `etcdTag` | +| `KUBE_OVN_VERSION` | Kube-OVN chart version pinned on every `Cluster` | `v4.4.6` | `kube-ovn (chart)` column of the same row; on an installed `global`, the `cpaas.io/kube-ovn-version` annotation of `Cluster/global` | +| `DCS_KUBERNETES_FILES_SECRET` | Provider-shipped Secret holding the admission, audit, and kubelet patch files | `dcs-kubernetes-1.35-files` | `kubectl -n cpaas-system get secret -o name \| grep dcs-kubernetes-` once the DCS provider runs | +| `LOCAL_REGISTRY_ADDRESS` | Bootstrap registry as seen from the bootstrap node, for package uploads | `127.0.0.1:11443` | Fixed value from the product page | +| `BOOTSTRAP_REGISTRY_ADDRESS` | Bootstrap registry as seen from inside the bootstrap cluster, for the AppRelease `repoURL` | `172.18.0.1:11443` | Fixed value from the product page | +| `INGRESS_CLASS_NAME` | Ingress class value in the provider AppReleases | `global-alb2` | Fixed value from Step 1 of the product page; keep the value the provider release requires | +| `KUBEADM_PROVIDER_VERSION` | Kubeadm provider chart version | `v1.0.14` | Tag list printed in [Install the provider packages](#install-the-provider-packages) | +| `DCS_PROVIDER_VERSION` | DCS provider chart version | `v1.0.22` | Same | +| `PROVIDER_SECRET_NAME` | DCS credential Secret on each bootstrap cluster and, after the import, on each `global`; every `DCSCluster` references it | `global-secret` | Product default | +| `PRIMARY_CLUSTER_VIP` | Self-built VIP of the primary `global` | `192.0.2.10` | Network plan | +| `STANDBY_CLUSTER_VIP` | Self-built VIP of the standby `global` | `203.0.113.10` | Network plan | +| `DCS_ENCRYPTION_PROVIDER_SECRET` | Secret holding the shared `encryption-provider.conf` | `encryption-provider-config` | Product default | +| `ENCRYPTION_PROVIDER_CONF` | Path of the shared encryption file on both bootstrap nodes, outside the delivery directory | `/root/dr/encryption-provider.conf` | Chosen by the delivery | + +Cluster files — DCS lines, the same names in every file: + +| Variable | Meaning | Example | Where to get it | +| --- | --- | --- | --- | +| `NODE_NAME_PREFIX` | Prefix of node hostnames and DCS VM names. Unique per cluster, so the two `global` sides never create VMs with the same name | `global-primary`, `global-standby`, `wl-prod` | Chosen. Workload files use `"${CLUSTER_NAME}"` | +| `HOST_IP` | Static IP of this side's bootstrap node (`global` files only) | `192.0.2.5` | Bootstrap node | +| `BOOTSTRAP_KUBECONFIG` | Kubeconfig of this side's bootstrap cluster (`global` files only) | `/root/.kube/config` | Written by Step 2 in [Bring up the bootstrap nodes](#bring-up-the-bootstrap-nodes) on that side's bootstrap node | +| `NODE_REGISTRY_ADDRESS` | Registry that provisioned nodes pull from, written to `cpaas.io/registry-address` | `global`: `"${HOST_IP}:11443"`; workload: `acp.example.com:11443` | `global`: the bootstrap registry. Workload: [read it from `global`](#read-the-shared-values-from-global) | +| `DCS_API_ENDPOINT` | DCS REST API URL, including the port | `https://192.0.2.200:7443` | DCS administrator. The API port is normally `7443`, not the `8443` web portal | +| `DCS_SITE` | DCS site ID | `A1B2C3D4` | DCS administrator | +| `DCS_HOST_MGMT_IPS` | Management IPs of every DCS host that may receive a clone, space-separated | `192.0.2.201 192.0.2.202 192.0.2.203` | DCS administrator | +| `DCS_HOST_MGMT_PORT` | File-upload port that DCS returns for those hosts | `8443` | DCS administrator | +| `DCS_VM_TEMPLATE` | VM template built from the Alauda OS `qcow2` image | `alauda-os-acp-4.4` | Created in [Collect the DCS facts](#collect-the-dcs-facts) | +| `DCS_VM_FOLDER` | Existing DCS VM folder for the cluster's VMs | `acp-global` | DCS administrator | +| `DCS_DATASTORE` | Datastore for every disk of the cluster | `datastore-01` | DCS administrator | +| `DCS_DVSWITCH` | Distributed virtual switch of the node NIC | `dvs-01` | DCS administrator | +| `DCS_PORT_GROUP` | Port group of the node NIC | `pg-acp-nodes` | DCS administrator | +| `DCS_VRID` | Keepalived VRID of the Self-built VIP, `1` to `255`, unique in the Layer 2 network | `51` | Network plan | +| `DCS_CONTROL_PLANE_HA` | `true` to let the provider keep the control-plane VMs on different hosts | `true` | `false` when DRS is not enabled on the DCS compute cluster | +| `DCS_NODE_MASK` | Prefix length of the node network, without `/` | `24` | Network plan | +| `DCS_NODE_GATEWAY` | Gateway of the node network | `192.0.2.1` | Network plan | +| `DCS_NODE_DNS` | DNS servers of the nodes, `;`-separated | `192.0.2.2` | Network plan | +| `CP_NODE_IP_1` | First control-plane node IP; `CP_NODE_IP_2` and `CP_NODE_IP_3` likewise | `192.0.2.11` | Network plan | +| `INFRA_NODE_IP_1` | First `infra` node IP; `INFRA_NODE_IP_2` and `INFRA_NODE_IP_3` likewise | `192.0.2.14` | Network plan | +| `DCS_CP_CPU` | Control-plane VM CPU cores | `16` | [Size the clusters](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#size-the-clusters) | +| `DCS_CP_MEMORY_MB` | Control-plane VM memory in MB | `32768` | Same | +| `DCS_CP_CPAAS_DISK_GB` | Persistent `/var/cpaas` disk of each control-plane node, in GB | `100` | Same | +| `DCS_INFRA_CPU` | `infra` VM CPU cores | `16` | Same | +| `DCS_INFRA_MEMORY_MB` | `infra` VM memory in MB | `32768` | Same | +| `DCS_INFRA_CPAAS_DISK_GB` | Persistent `/var/cpaas` disk of each `infra` node, in GB | `500` | Same | + +The dedicated-pool layout adds the `infra` values once per dedicated pool — `INGRESS_NODE_IP_1` to `_3`, `DCS_INGRESS_CPU`, `DCS_INGRESS_MEMORY_MB`, `DCS_INGRESS_CPAAS_DISK_GB`, and the same for `MONITOR` and `LOG`; see [Dedicated pools](#dedicated-pools). A worker pool adds the same set with the `WORKER` prefix; see [Worker pools](#worker-pools). + +Set at run time, never written to a file (in addition to the common article's runtime secrets `ACP_TOKEN`, `REGISTRY_USERNAME`, and `REGISTRY_PASSWORD`): + +| Variable | Meaning | Example | Where to get it | +| --- | --- | --- | --- | +| `DCS_AUTH_USER` | DCS API user | (never stored) | Prompted by `read` | +| `DCS_AUTH_KEY` | DCS API password | (never stored) | Prompted by `read` | +| `DCS_PROVIDER_PACK` | Path of the downloaded DCS provider package | (set by the command) | Set from `ls` | +| `KUBEADM_PROVIDER_PACK` | Path of the downloaded Kubeadm provider package | (set by the command) | Set from `ls` | +| `ENCRYPTION_PROVIDER_SECRET_B64` | Random 32-byte encryption key, base64-encoded | (never stored) | Generated, then unset | +| `INSTALLER_IP` | Pod IP of `cpaas-installer` on this bootstrap node | (set by the command) | Set from `kubectl` | +| `PLATFORM_ADMIN_PASSWORD_B64` | Base64-encoded platform administrator password for the installer request | (never stored) | Prompted by `read` | +| `REGISTRY_PASSWORD_B64` | Base64-encoded platform registry password for the installer request | (never stored) | Prompted by `read` | +| `ACTIVE_CLUSTER_TOKEN` | Bearer token of the primary's `k8sadmin` ServiceAccount | (never stored) | Read from the primary into the shell | + +`site.env` — append the DCS lines to the common lines: + +```bash +# site.env: DCS lines +IMAGE_REPOSITORY='cloud.alauda.io/alauda' +DNS_IMAGE_TAG='1.14.3-v4.4.6' +ETCD_IMAGE_TAG='v3.6.13-acp.0' +KUBE_OVN_VERSION='v4.4.6' +DCS_KUBERNETES_FILES_SECRET='dcs-kubernetes-1.35-files' +LOCAL_REGISTRY_ADDRESS='127.0.0.1:11443' +BOOTSTRAP_REGISTRY_ADDRESS='172.18.0.1:11443' +INGRESS_CLASS_NAME='global-alb2' +KUBEADM_PROVIDER_VERSION='v1.0.14' +DCS_PROVIDER_VERSION='v1.0.22' +PROVIDER_SECRET_NAME='global-secret' +# Disaster recovery: read by both global sides, so the increments are identical. +PRIMARY_CLUSTER_VIP='192.0.2.10' +STANDBY_CLUSTER_VIP='203.0.113.10' +DCS_ENCRYPTION_PROVIDER_SECRET='encryption-provider-config' +ENCRYPTION_PROVIDER_CONF='/root/dr/encryption-provider.conf' +``` + +`global-primary.env` — the common lines that carry DCS-specific values, then the DCS lines: + +```bash +# global-primary.env +CLUSTER_NAME='global' +CONTROL_PLANE_VIP="${PRIMARY_CLUSTER_VIP}" +CLUSTER_CIDR='100.3.0.0/16' +SERVICE_CIDR='100.4.0.0/16' +KUBE_OVN_JOIN_CIDR='100.5.0.0/16' +GLOBAL_KUBECONFIG='/root/.kube/acp/global-primary.kubeconfig' +CLUSTER_KUBECONFIG='/root/.kube/acp/global-primary.kubeconfig' +# ... the other common lines (monitoring, logging, ingress) as in the common article + +# DCS lines +NODE_NAME_PREFIX='global-primary' +HOST_IP='192.0.2.5' +BOOTSTRAP_KUBECONFIG='/root/.kube/config' +NODE_REGISTRY_ADDRESS="${HOST_IP}:11443" +DCS_API_ENDPOINT='https://192.0.2.200:7443' +DCS_SITE='A1B2C3D4' +DCS_HOST_MGMT_IPS='192.0.2.201 192.0.2.202 192.0.2.203' +DCS_HOST_MGMT_PORT='8443' +DCS_VM_TEMPLATE='alauda-os-acp-4.4' +DCS_VM_FOLDER='acp-global' +DCS_DATASTORE='datastore-01' +DCS_DVSWITCH='dvs-01' +DCS_PORT_GROUP='pg-acp-nodes' +DCS_VRID='51' +DCS_CONTROL_PLANE_HA='true' +DCS_NODE_MASK='24' +DCS_NODE_GATEWAY='192.0.2.1' +DCS_NODE_DNS='192.0.2.2' +CP_NODE_IP_1='192.0.2.11' +CP_NODE_IP_2='192.0.2.12' +CP_NODE_IP_3='192.0.2.13' +INFRA_NODE_IP_1='192.0.2.14' +INFRA_NODE_IP_2='192.0.2.15' +INFRA_NODE_IP_3='192.0.2.16' +DCS_CP_CPU='16' +DCS_CP_MEMORY_MB='32768' +DCS_CP_CPAAS_DISK_GB='100' +DCS_INFRA_CPU='16' +DCS_INFRA_MEMORY_MB='32768' +DCS_INFRA_CPAAS_DISK_GB='500' +``` + +`global-standby.env` is a copy of `global-primary.env` with the standby's lines changed. When the standby runs in a second DCS site, every `DCS_` fact changes as well: + +```bash +# global-standby.env: differences from global-primary.env +CONTROL_PLANE_VIP="${STANDBY_CLUSTER_VIP}" +CLUSTER_CIDR='100.13.0.0/16' +SERVICE_CIDR='100.14.0.0/16' +KUBE_OVN_JOIN_CIDR='100.15.0.0/16' +GLOBAL_KUBECONFIG='/root/.kube/acp/global-standby.kubeconfig' +CLUSTER_KUBECONFIG='/root/.kube/acp/global-standby.kubeconfig' +NODE_NAME_PREFIX='global-standby' +HOST_IP='203.0.113.5' +DCS_VRID='52' +DCS_NODE_GATEWAY='203.0.113.1' +DCS_NODE_DNS='203.0.113.2' +CP_NODE_IP_1='203.0.113.11' +CP_NODE_IP_2='203.0.113.12' +CP_NODE_IP_3='203.0.113.13' +INFRA_NODE_IP_1='203.0.113.14' +INFRA_NODE_IP_2='203.0.113.15' +INFRA_NODE_IP_3='203.0.113.16' +``` + +`wl-prod.env` — the DCS lines that follow the common lines of the workload cluster (`CLUSTER_NAME='wl-prod'`, `CONTROL_PLANE_VIP='192.0.2.20'`, and so on): + +```bash +# wl-prod.env (continued): DCS lines +NODE_NAME_PREFIX="${CLUSTER_NAME}" +NODE_REGISTRY_ADDRESS='acp.example.com:11443' +DCS_API_ENDPOINT='https://192.0.2.200:7443' +DCS_SITE='A1B2C3D4' +DCS_HOST_MGMT_IPS='192.0.2.201 192.0.2.202 192.0.2.203' +DCS_HOST_MGMT_PORT='8443' +DCS_VM_TEMPLATE='alauda-os-acp-4.4' +DCS_VM_FOLDER='acp-wl-prod' +DCS_DATASTORE='datastore-01' +DCS_DVSWITCH='dvs-01' +DCS_PORT_GROUP='pg-acp-nodes' +DCS_VRID='53' +DCS_CONTROL_PLANE_HA='true' +DCS_NODE_MASK='24' +DCS_NODE_GATEWAY='192.0.2.1' +DCS_NODE_DNS='192.0.2.2' +CP_NODE_IP_1='192.0.2.21' +CP_NODE_IP_2='192.0.2.22' +CP_NODE_IP_3='192.0.2.23' +INFRA_NODE_IP_1='192.0.2.24' +INFRA_NODE_IP_2='192.0.2.25' +INFRA_NODE_IP_3='192.0.2.26' +DCS_CP_CPU='8' +DCS_CP_MEMORY_MB='16384' +DCS_CP_CPAAS_DISK_GB='100' +DCS_INFRA_CPU='16' +DCS_INFRA_MEMORY_MB='32768' +DCS_INFRA_CPAAS_DISK_GB='500' +``` + +### Prepare DCS + +#### Collect the DCS facts + +Collect these with the DCS administrator before anything is installed, and record each in the cluster file named in the last column. + +| Fact | Requirement | Variable | +| --- | --- | --- | +| REST API endpoint | The VRM address and the API port, normally `7443`. The `8443` web portal is for people, not for the provider | `DCS_API_ENDPOINT` | +| API user | Role `administrator`. An interface interconnection user (the default) needs the DCS policy "Whether to modify the password of an interface interconnection user upon password resetting and first login" set to **No**; otherwise the first login forces a password change and provisioning fails. A domain user needs DCS domain authentication and the user pre-created with the `administrator` role. See [Credential User Types](https://docs.alauda.io/immutable-infra/1.0/infrastructure/huawei-dcs.html#user-types) | Prompted, never stored | +| Site ID | All resources of a cluster live in one site | `DCS_SITE` | +| VM template | Built from the Alauda OS `qcow2` image published for the installed ACP 4.4 release ([Downloading Alauda OS](https://docs.alauda.io/immutable-infra/1.0/overview/os-support-matrix.html#downloading-alauda-os)). When the DR pair spans two DCS sites, create a template in each. The DCS hosts must expose the `x86-64-v2` CPU baseline to the VMs | `DCS_VM_TEMPLATE` | +| VM folder | Exists before the cluster is created | `DCS_VM_FOLDER` | +| Compute cluster | DRS enabled and at least three usable hosts, if `DCS_CONTROL_PLANE_HA=true` | `DCS_CONTROL_PLANE_HA` | +| Datastore | Free capacity for the system disk, the template disks, and `/var/cpaas` of every node, kept below roughly 70 % used; visible to every host that may run the VMs; accepts direct file upload for the Ignition ISO (otherwise provide shared storage such as NFS). Use the read-only checks in [DCS Platform Capacity and Placement](https://docs.alauda.io/immutable-infra/1.0/infrastructure/huawei-dcs.html#capacity-and-placement) | `DCS_DATASTORE` | +| DVS and port group | Carry the node NIC; allow VIP ownership, GARP/ARP updates, and VRRP between the control-plane VMs | `DCS_DVSWITCH`, `DCS_PORT_GROUP` | +| Host management IPs | Every host that may receive a clone, and the upload port DCS returns for them (typically `8443`) | `DCS_HOST_MGMT_IPS`, `DCS_HOST_MGMT_PORT` | +| Node network | Node IPs per pool, prefix length, gateway, DNS servers | `CP_NODE_IP_n`, `INFRA_NODE_IP_n`, `DCS_NODE_MASK`, `DCS_NODE_GATEWAY`, `DCS_NODE_DNS` | +| Control-plane VIP | Unused IPv4 address in the control-plane Layer 2 network, outside every pool, and a VRID unique in that network | `CONTROL_PLANE_VIP`, `DCS_VRID` | + +For the concepts behind these names — a DCS Cluster is a host pool, not the `DCSCluster` resource — see [Huawei DCS Concepts and Terminology](https://docs.alauda.io/immutable-infra/1.0/overview/providers/huawei-dcs.html#concepts-and-terminology). + +#### Open the network paths + +These are the DCS-specific flows. Have them approved and tested before you install. For the platform's own port inventory and the flows it lists separately, use the [Communication Matrix](https://docs.alauda.io/container_platform/4.4/install/prepare/communication_matrix.html). + +| From | To | Port | Why | +| --- | --- | --- | --- | +| Bootstrap node | DCS VRM (`DCS_API_ENDPOINT`) | TCP `7443` | The providers run in the bootstrap cluster while `global` is installed: REST API calls and the first step of every file upload (`applyUpload`) | +| Bootstrap node | Every DCS host management IP | `DCS_HOST_MGMT_PORT`, typically TCP `8443` | The Ignition ISO is streamed straight to the host that the `applyUpload` response names. A blocked host shows up as a VM that boots without Ignition and never registers | +| Bootstrap node | `global` control-plane VIP | TCP `6443` | Cluster API reaches the new control plane | +| `global` nodes | Bootstrap node | TCP `11443` | Image pulls until the `global` registry takes over | +| `global` nodes | DCS VRM and every host management IP | TCP `7443` and `DCS_HOST_MGMT_PORT` | After the installation the providers run on `global`: workload clusters, scale-out, and node replacement use the same two paths. On multi-NIC nodes, both must route through the right NIC | +| Control-plane nodes of each cluster | Each other | VRRP (IP protocol 112) and GARP | Self-built VIP | +| `global` nodes | Workload control-plane VIP and control-plane nodes | TCP `6443`, `11780` | `global` manages the workload clusters; see [Connecting global Cluster and Workload Cluster](https://docs.alauda.io/container_platform/4.4/configure/clusters/on-premises.html#connecting-global-cluster-and-workload-cluster) | +| Primary `global` network | Standby `global` network, both directions | TCP `443`, `6443`, `11443`, `2379` (`80` only for HTTP access) | DR; see [Network Requirements](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#network-requirements) | + +Check the DCS paths from each bootstrap node: + +```bash +# on the bootstrap node, in the delivery directory +load_env global-primary.env +curl -k -sS -o /dev/null --connect-timeout 5 -w 'VRM API: HTTP %{http_code}\n' "${DCS_API_ENDPOINT}" +for ip in ${DCS_HOST_MGMT_IPS}; do + timeout 5 bash -c "` with the Core Package file: + +```bash +# on the bootstrap node +mkdir -p /root/cpaas-install +tar -xvf -C /root/cpaas-install +cd /root/cpaas-install/installer +bash setup.sh +mkdir -p "$HOME/.local/bin" ~/.kube +if ! command -v kubectl >/dev/null 2>&1; then + nerdctl cp minialauda-control-plane:/usr/bin/kubectl "$HOME/.local/bin/kubectl" + chmod 0755 "$HOME/.local/bin/kubectl" + export PATH="$HOME/.local/bin:$PATH" +fi +cp /var/cpaas/data/alauda.kubeconfig ~/.kube/config +kubectl get nodes +umask 077; mkdir -p /root/.kube/acp # for the global kubeconfigs of the common Kubeconfig convention +``` + +`/root/.kube/config` is the `BOOTSTRAP_KUBECONFIG` of this side's cluster file; every `# on the bootstrap cluster` command passes it explicitly, so a kubeconfig of another cluster in the environment cannot redirect it. + +#### Install the provider packages + +Upload the Kubeadm and DCS provider packages (placed under `/root/packages/` here) to the bootstrap registry, and read the chart versions they carry: + +```bash +# on the bootstrap node, in the delivery directory +load_env global-primary.env +DCS_PROVIDER_PACK=$(ls /root/packages/cluster-api-provider-dcs.amd64.*.tgz) +KUBEADM_PROVIDER_PACK=$(ls /root/packages/cluster-api-provider-kubeadm.amd64.*.tgz) +/root/cpaas-install/installer/res/amd64/packtool pack push -r "${LOCAL_REGISTRY_ADDRESS}" -c "${DCS_PROVIDER_PACK}" +/root/cpaas-install/installer/res/amd64/packtool pack push -r "${LOCAL_REGISTRY_ADDRESS}" -c "${KUBEADM_PROVIDER_PACK}" +curl -s "http://${LOCAL_REGISTRY_ADDRESS}/v2/ait/chart-cluster-api-provider-kubeadm/tags/list"; echo +curl -s "http://${LOCAL_REGISTRY_ADDRESS}/v2/ait/chart-cluster-api-provider-dcs/tags/list"; echo +``` + +Record the two tags as `KUBEADM_PROVIDER_VERSION` and `DCS_PROVIDER_VERSION` in `site.env` — both bootstrap nodes use the same packages, so the values are shared — and reload it. + +`global-primary/00-provider-appreleases.yaml` — the provider AppReleases of Step 3 of the product page, for the default anonymous bootstrap registry. `global.cluster.type: Baremetal` is a chart-internal classifier and stays `Baremetal` on DCS: + +```yaml +apiVersion: operator.alauda.io/v1alpha1 +kind: AppRelease +metadata: + annotations: + auto-recycle: "true" + interval-sync: "true" + name: cluster-api-provider-kubeadm + namespace: cpaas-system +spec: + destination: + cluster: "" + namespace: "" + source: + charts: + - name: ait/chart-cluster-api-provider-kubeadm + releaseName: cluster-api-provider-kubeadm + targetRevision: ${KUBEADM_PROVIDER_VERSION} + repoURL: ${BOOTSTRAP_REGISTRY_ADDRESS} + timeout: 120 + values: + global: + albName: ${INGRESS_CLASS_NAME} + auth: + default_admin: admin@cpaas.io + cluster: + isGlobal: true + name: global + networkType: kube-ovn + type: Baremetal + host: ${PLATFORM_HOST} + ingress: + ingressClassName: ${INGRESS_CLASS_NAME} + labelBaseDomain: cpaas.io + namespace: cpaas-system + platformUrl: https://${PLATFORM_HOST} + protectSecretFiles: + enabled: false + region: global + registry: + address: ${BOOTSTRAP_REGISTRY_ADDRESS} + replicas: 1 + scheme: https +--- +apiVersion: operator.alauda.io/v1alpha1 +kind: AppRelease +metadata: + annotations: + auto-recycle: "true" + interval-sync: "true" + name: cluster-api-provider-dcs + namespace: cpaas-system +spec: + destination: + cluster: "" + namespace: "" + source: + charts: + - name: ait/chart-cluster-api-provider-dcs + releaseName: cluster-api-provider-dcs + targetRevision: ${DCS_PROVIDER_VERSION} + repoURL: ${BOOTSTRAP_REGISTRY_ADDRESS} + timeout: 120 + values: + global: + albName: ${INGRESS_CLASS_NAME} + auth: + default_admin: admin@cpaas.io + cluster: + isGlobal: true + name: global + networkType: kube-ovn + type: Baremetal + host: ${PLATFORM_HOST} + ingress: + ingressClassName: ${INGRESS_CLASS_NAME} + labelBaseDomain: cpaas.io + namespace: cpaas-system + platformUrl: https://${PLATFORM_HOST} + protectSecretFiles: + enabled: false + region: global + registry: + address: ${BOOTSTRAP_REGISTRY_ADDRESS} + replicas: 1 + scheme: https +``` + +If you set a registry user and password during the bootstrap setup, `setup.sh` created the `global-registry-auth` Secret; then add `spec.source.chartPullSecret: global-registry-auth` and `spec.values.global.registry.imagePullSecrets: [global-registry-auth]` to both AppReleases, as the product page shows. With the default anonymous registry, add neither. + +```bash +# on the bootstrap cluster, in the delivery directory +load_env global-primary.env +apply_rendered global-primary.env global-primary/00-provider-appreleases.yaml "${BOOTSTRAP_KUBECONFIG}" +until kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" get crd kubeadmcontrolplanes.controlplane.cluster.x-k8s.io >/dev/null 2>&1; do sleep 10; done +until kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" get crd dcsclusters.infrastructure.cluster.x-k8s.io >/dev/null 2>&1; do sleep 10; done +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" get pods -n cpaas-system | grep -E 'kubeadm|dcs' +for k in control-plane-kubelet-patch worker-kubelet-patch; do + kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get secret "${DCS_KUBERNETES_FILES_SECRET}" -o jsonpath="{.data.$k\.json}" \ + | base64 -d | grep -c imagePullCredentialsVerificationPolicy +done +``` + +Pass: both CRDs exist, the provider Pods are `Running`, and the loop prints `1` twice — both provider-shipped kubelet patches carry the setting that Kubernetes 1.35 requires. If the Secret is missing, replace the `contentFrom` entries of `12-kcp.yaml` and `22-infra-kct.yaml` with the inline `content` of the [Complete KubeadmControlPlane Configuration](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-dcs.html#complete-kubeadmcontrolplane-configuration) and of the [worker bootstrap template](https://docs.alauda.io/immutable-infra/1.0/manage-nodes/huawei-dcs.html#worker-node-deployment). + +#### Create the DCS credential Secret + +Create the Secret on each bootstrap cluster before its manifests are applied. The user and password pass through process substitution only, so they never reach a disk, the shell history, or a command line: + +```bash +# on the bootstrap cluster +read -rp 'DCS API user: ' DCS_AUTH_USER +read -rsp 'DCS API password: ' DCS_AUTH_KEY; echo +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system create secret generic "${PROVIDER_SECRET_NAME}" \ + --from-file=authUser=<(printf '%s' "${DCS_AUTH_USER}") \ + --from-file=authKey=<(printf '%s' "${DCS_AUTH_KEY}") \ + --from-literal=endpoint="${DCS_API_ENDPOINT}" \ + --from-literal=site="${DCS_SITE}" +unset DCS_AUTH_USER DCS_AUTH_KEY +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system label secret "${PROVIDER_SECRET_NAME}" --overwrite cpaas.io/cluster-name=global +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get secret "${PROVIDER_SECRET_NAME}" -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}' +``` + +The last command lists the keys only: `authKey`, `authUser`, `endpoint`, `site`. For a domain user, add `--from-literal=userType=domain`; without that key the provider authenticates as an interface interconnection user. To rotate the password later, patch the Secret as described in [Cloud Credentials](https://docs.alauda.io/immutable-infra/1.0/infrastructure/huawei-dcs.html#using-yaml) — never re-apply a manifest. + +### Install the global cluster + +This section installs one `global` side from its bootstrap node. It is written for the primary with `global-primary.env`; [Add disaster recovery](#add-disaster-recovery) runs it a second time for the standby. The manifests carry the three DR increments, marked `# DR increment`, because the multi-cluster line always has a standby: complete [Prepare the shared DR inputs](#prepare-the-shared-dr-inputs) before you apply them. For a single `global` without DR, see [Single cluster](#single-cluster). + +The resources follow Step 4 of [Installing the global Cluster](https://docs.alauda.io/immutable-infra/1.0/global/install.html) and its [complete DCS worked example](https://docs.alauda.io/immutable-infra/1.0/global/install.html#dcs-worked-example) with the Self-built VIP variant, the `kubeadmConfigSpec` body of the [Complete KubeadmControlPlane Configuration](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-dcs.html#complete-kubeadmcontrolplane-configuration), and an `infra` pool built from [Worker Node Deployment](https://docs.alauda.io/immutable-infra/1.0/manage-nodes/huawei-dcs.html#worker-node-deployment). + +#### Write the manifests + +`global-primary/10-cp-pool.yaml` — the control-plane IPs, hostnames, and the persistent `/var/cpaas` disk of each slot: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: DCSIpHostnamePool +metadata: + name: global-cp-pool + namespace: cpaas-system + labels: + cpaas.io/cluster-name: "global" +spec: + pool: + - ip: "${CP_NODE_IP_1}" + mask: "${DCS_NODE_MASK}" + gateway: "${DCS_NODE_GATEWAY}" + dns: "${DCS_NODE_DNS}" + hostname: "${NODE_NAME_PREFIX}-cp-1" + machineName: "${NODE_NAME_PREFIX}-cp-1" + persistentDisk: + - slot: 0 + quantityGB: ${DCS_CP_CPAAS_DISK_GB} + datastoreName: "${DCS_DATASTORE}" + path: /var/cpaas + format: xfs + mountOptions: + - defaults + - ip: "${CP_NODE_IP_2}" + mask: "${DCS_NODE_MASK}" + gateway: "${DCS_NODE_GATEWAY}" + dns: "${DCS_NODE_DNS}" + hostname: "${NODE_NAME_PREFIX}-cp-2" + machineName: "${NODE_NAME_PREFIX}-cp-2" + persistentDisk: + - slot: 0 + quantityGB: ${DCS_CP_CPAAS_DISK_GB} + datastoreName: "${DCS_DATASTORE}" + path: /var/cpaas + format: xfs + mountOptions: + - defaults + - ip: "${CP_NODE_IP_3}" + mask: "${DCS_NODE_MASK}" + gateway: "${DCS_NODE_GATEWAY}" + dns: "${DCS_NODE_DNS}" + hostname: "${NODE_NAME_PREFIX}-cp-3" + machineName: "${NODE_NAME_PREFIX}-cp-3" + persistentDisk: + - slot: 0 + quantityGB: ${DCS_CP_CPAAS_DISK_GB} + datastoreName: "${DCS_DATASTORE}" + path: /var/cpaas + format: xfs + mountOptions: + - defaults +``` + +`global-primary/11-cp-template.yaml` — the control-plane VM. The four template disks are mandatory for a control-plane template; `/var/cpaas` is deliberately not one of them: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: DCSMachineTemplate +metadata: + name: global-cp-template + namespace: cpaas-system + labels: + cpaas.io/cluster-name: "global" +spec: + template: + spec: + vmTemplateName: "${DCS_VM_TEMPLATE}" + location: + type: folder + name: "${DCS_VM_FOLDER}" + vmConfig: + dvSwitchName: "${DCS_DVSWITCH}" + portGroupName: "${DCS_PORT_GROUP}" + dcsMachineCpuSpec: + quantity: ${DCS_CP_CPU} + dcsMachineMemorySpec: # MB + quantity: ${DCS_CP_MEMORY_MB} + dcsMachineDiskSpec: # GB; 0 keeps the size of the VM template's system disk + - quantity: 0 + datastoreName: "${DCS_DATASTORE}" + systemVolume: true + - quantity: 10 + datastoreName: "${DCS_DATASTORE}" + path: /var/lib/etcd + format: xfs + - quantity: 100 + datastoreName: "${DCS_DATASTORE}" + path: /var/lib/kubelet + format: xfs + - quantity: 100 + datastoreName: "${DCS_DATASTORE}" + path: /var/lib/containerd + format: xfs + ipHostPoolRef: + name: global-cp-pool +``` + +`global-primary/12-kcp.yaml` — the control plane. Compared with a workload cluster, `global` keeps only the `skip-kube-proxy` annotation and adds etcd server certificate SANs. The three large files come from the provider-shipped Secret: + +```yaml +apiVersion: controlplane.cluster.x-k8s.io/v1beta1 +kind: KubeadmControlPlane +metadata: + name: global-kcp + namespace: cpaas-system + labels: + cpaas.io/cluster-name: "global" + annotations: + controlplane.cluster.x-k8s.io/skip-kube-proxy: "" +spec: + replicas: 3 + version: ${K8S_VERSION} + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + machineTemplate: + nodeDrainTimeout: 1m + nodeDeletionTimeout: 5m + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: DCSMachineTemplate + name: global-cp-template + kubeadmConfigSpec: + format: ignition + users: + - name: boot + sshAuthorizedKeys: + - "${BOOT_SSH_PUBLIC_KEY}" + files: + - path: /etc/kubernetes/admission/psa-config.yaml + owner: "root:root" + permissions: "0644" + contentFrom: + secret: + name: ${DCS_KUBERNETES_FILES_SECRET} + key: psa-config.yaml + - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json + owner: "root:root" + permissions: "0644" + contentFrom: + secret: + name: ${DCS_KUBERNETES_FILES_SECRET} + key: control-plane-kubelet-patch.json + - path: /etc/kubernetes/audit/policy.yaml + owner: "root:root" + permissions: "0644" + contentFrom: + secret: + name: ${DCS_KUBERNETES_FILES_SECRET} + key: audit-policy.yaml + preKubeadmCommands: + - while ! ip route | grep -q "default via"; do sleep 1; done; echo "NetworkManager started" + - mkdir -p /run/cluster-api && restorecon -Rv /run/cluster-api + - if [ -f /etc/disk-setup.sh ]; then bash /etc/disk-setup.sh; fi + postKubeadmCommands: + - chmod 600 /var/lib/kubelet/config.yaml + clusterConfiguration: + imageRepository: ${IMAGE_REPOSITORY} + dns: + imageTag: "${DNS_IMAGE_TAG}" + etcd: + local: + imageTag: "${ETCD_IMAGE_TAG}" + serverCertSANs: # DR increment: the same list on both sides + - "${PRIMARY_CLUSTER_VIP}" + - "${STANDBY_CLUSTER_VIP}" + - "${PLATFORM_HOST}" + - "etcd.kube-system" + apiServer: + extraArgs: + audit-log-format: json + audit-log-maxage: "30" + audit-log-maxbackup: "10" + audit-log-maxsize: "200" + profiling: "false" + audit-log-mode: batch + audit-log-path: /etc/kubernetes/audit/audit.log + audit-policy-file: /etc/kubernetes/audit/policy.yaml + tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384" + encryption-provider-config: /etc/kubernetes/encryption-provider.conf + admission-control-config-file: /etc/kubernetes/admission/psa-config.yaml + tls-min-version: VersionTLS12 + kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt + extraVolumes: + - name: vol-dir-0 + hostPath: /etc/kubernetes + mountPath: /etc/kubernetes + pathType: Directory + controllerManager: + extraArgs: + bind-address: "::" + profiling: "false" + tls-min-version: VersionTLS12 + flex-volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" + scheduler: + extraArgs: + bind-address: "::" + tls-min-version: VersionTLS12 + profiling: "false" + initConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + node-labels: "kube-ovn/role=master" + provider-id: PROVIDER_ID + volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" + protect-kernel-defaults: "true" + joinConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + node-ip: NODE_IP + node-labels: "kube-ovn/role=master" + provider-id: PROVIDER_ID + volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" + protect-kernel-defaults: "true" +``` + +Keep `encryption-provider-config`: the provider writes the file it points at, and removing the argument turns encryption at rest off. + +`global-primary/13-dcscluster.yaml` — the DCS infrastructure cluster with the Self-built VIP: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: DCSCluster +metadata: + name: global + namespace: cpaas-system + labels: + cpaas.io/cluster-name: "global" + annotations: + cpaas.io/registry-address: "${NODE_REGISTRY_ADDRESS}" +spec: + controlPlaneLoadBalancer: + host: "${CONTROL_PLANE_VIP}" + port: 6443 + type: internal + vrid: ${DCS_VRID} + controlPlaneEndpoint: + host: "${CONTROL_PLANE_VIP}" + port: 6443 + credentialSecretRef: + name: "${PROVIDER_SECRET_NAME}" + encryptionProviderConfigRef: # DR increment + name: "${DCS_ENCRYPTION_PROVIDER_SECRET}" + controlPlaneHA: + enabled: ${DCS_CONTROL_PLANE_HA} + networkType: kube-ovn + site: "${DCS_SITE}" +``` + +`controlPlaneLoadBalancer.interface` is left unset, so the provider detects the NIC from the node IP. Set it (for example to `eth1`) only on multi-NIC nodes where the VIP must sit on a specific interface — it cannot be changed later. + +`global-primary/14-cluster.yaml` — the Cluster API `Cluster` with the `global` labels and annotations. The installer later rewrites `cpaas.io/registry-address` on this object and on the `DCSCluster` to the permanent platform registry: + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: global + namespace: cpaas-system + labels: + cpaas.io/cluster-name: "global" + cluster-type: DCS + is-global: "true" + annotations: + capi.cpaas.io/resource-group-version: infrastructure.cluster.x-k8s.io/v1beta1 + capi.cpaas.io/resource-kind: DCSCluster + capi.cpaas.io/kubernetes: "${K8S_VERSION}" + cpaas.io/registry-address: "${NODE_REGISTRY_ADDRESS}" + cpaas.io/nodes-mode: self-managed + cpaas.io/kube-ovn-join-cidr: "${KUBE_OVN_JOIN_CIDR}" + cpaas.io/kube-ovn-version: "${KUBE_OVN_VERSION}" +spec: + clusterNetwork: + pods: + cidrBlocks: + - "${CLUSTER_CIDR}" + services: + cidrBlocks: + - "${SERVICE_CIDR}" + controlPlaneRef: + apiVersion: controlplane.cluster.x-k8s.io/v1beta1 + kind: KubeadmControlPlane + name: global-kcp + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: DCSCluster + name: global +``` + +`global-primary/20-infra-pool.yaml` — the `infra` pool slots: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: DCSIpHostnamePool +metadata: + name: global-infra-pool + namespace: cpaas-system + labels: + cpaas.io/cluster-name: "global" +spec: + pool: + - ip: "${INFRA_NODE_IP_1}" + mask: "${DCS_NODE_MASK}" + gateway: "${DCS_NODE_GATEWAY}" + dns: "${DCS_NODE_DNS}" + hostname: "${NODE_NAME_PREFIX}-infra-1" + machineName: "${NODE_NAME_PREFIX}-infra-1" + persistentDisk: + - slot: 0 + quantityGB: ${DCS_INFRA_CPAAS_DISK_GB} + datastoreName: "${DCS_DATASTORE}" + path: /var/cpaas + format: xfs + mountOptions: + - defaults + - ip: "${INFRA_NODE_IP_2}" + mask: "${DCS_NODE_MASK}" + gateway: "${DCS_NODE_GATEWAY}" + dns: "${DCS_NODE_DNS}" + hostname: "${NODE_NAME_PREFIX}-infra-2" + machineName: "${NODE_NAME_PREFIX}-infra-2" + persistentDisk: + - slot: 0 + quantityGB: ${DCS_INFRA_CPAAS_DISK_GB} + datastoreName: "${DCS_DATASTORE}" + path: /var/cpaas + format: xfs + mountOptions: + - defaults + - ip: "${INFRA_NODE_IP_3}" + mask: "${DCS_NODE_MASK}" + gateway: "${DCS_NODE_GATEWAY}" + dns: "${DCS_NODE_DNS}" + hostname: "${NODE_NAME_PREFIX}-infra-3" + machineName: "${NODE_NAME_PREFIX}-infra-3" + persistentDisk: + - slot: 0 + quantityGB: ${DCS_INFRA_CPAAS_DISK_GB} + datastoreName: "${DCS_DATASTORE}" + path: /var/cpaas + format: xfs + mountOptions: + - defaults +``` + +`global-primary/21-infra-template.yaml` — the `infra` VM; worker templates need the system disk, `/var/lib/kubelet`, and `/var/lib/containerd`: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: DCSMachineTemplate +metadata: + name: global-infra-template + namespace: cpaas-system + labels: + cpaas.io/cluster-name: "global" +spec: + template: + spec: + vmTemplateName: "${DCS_VM_TEMPLATE}" + location: + type: folder + name: "${DCS_VM_FOLDER}" + vmConfig: + dvSwitchName: "${DCS_DVSWITCH}" + portGroupName: "${DCS_PORT_GROUP}" + dcsMachineCpuSpec: + quantity: ${DCS_INFRA_CPU} + dcsMachineMemorySpec: # MB + quantity: ${DCS_INFRA_MEMORY_MB} + dcsMachineDiskSpec: # GB + - quantity: 0 + datastoreName: "${DCS_DATASTORE}" + systemVolume: true + - quantity: 100 + datastoreName: "${DCS_DATASTORE}" + path: /var/lib/kubelet + format: xfs + - quantity: 100 + datastoreName: "${DCS_DATASTORE}" + path: /var/lib/containerd + format: xfs + ipHostPoolRef: + name: global-infra-pool +``` + +`global-primary/22-infra-kct.yaml` — the bootstrap template of the `infra` nodes: + +```yaml +apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 +kind: KubeadmConfigTemplate +metadata: + name: global-infra-kct + namespace: cpaas-system + labels: + cpaas.io/cluster-name: "global" +spec: + template: + spec: + format: ignition + users: + - name: boot + sshAuthorizedKeys: + - "${BOOT_SSH_PUBLIC_KEY}" + files: + - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json + owner: "root:root" + permissions: "0644" + contentFrom: + secret: + name: ${DCS_KUBERNETES_FILES_SECRET} + key: worker-kubelet-patch.json + preKubeadmCommands: + - while ! ip route | grep -q "default via"; do sleep 1; done; echo "NetworkManager started" + - mkdir -p /run/cluster-api && restorecon -Rv /run/cluster-api + - if [ -f /etc/disk-setup.sh ]; then bash /etc/disk-setup.sh; fi + postKubeadmCommands: + - chmod 600 /var/lib/kubelet/config.yaml + joinConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + node-ip: NODE_IP + provider-id: PROVIDER_ID + volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" + protect-kernel-defaults: "true" +``` + +`global-primary/23-infra-md.yaml` — the `infra` pool, with its role labels and taint declared on the pool, never on the nodes. `node-role.kubernetes.io/worker` puts the nodes into the default worker MachineConfigPool; the `infra` taint keeps everything that does not tolerate it off the pool: + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: MachineDeployment +metadata: + name: global-infra + namespace: cpaas-system + labels: + cpaas.io/cluster-name: "global" +spec: + clusterName: global + replicas: 3 + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + maxUnavailable: 1 + selector: + matchLabels: null + template: + metadata: + labels: + cluster.x-k8s.io/cluster-name: global + pool.name: global-infra + node-role.kubernetes.io/infra: "" + node-role.kubernetes.io/worker: "" + spec: + clusterName: global + version: ${K8S_VERSION} + nodeDrainTimeout: 1m + nodeDeletionTimeout: 5m + taints: + - key: node-role.kubernetes.io/infra + value: reserved + effect: NoSchedule + bootstrap: + configRef: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: global-infra-kct + namespace: cpaas-system + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: DCSMachineTemplate + name: global-infra-template + namespace: cpaas-system +``` + +`global-primary/30-import-extra-resources.yaml` — Step 7 of the product page. The installer copies the listed Secrets into the new `global`. The built-in DCS migration moves the Cluster API resources but no Secret, so a Secret missing from this list exists only in the bootstrap cluster, and the first scale-out or control-plane replacement after the bootstrap node is gone fails: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: dcs-import-extra-resources + namespace: cpaas-system +data: + resources.yaml: | + resources: + - resource: "secrets" + names: + - "${PROVIDER_SECRET_NAME}" + - "${DCS_ENCRYPTION_PROVIDER_SECRET}" # DR increment + method: kubectl +``` + +`global-primary/40-installer-config.json` — the installer request of Step 8. The two passwords and the registry user are placeholders that the trigger command fills in from prompts, so the file never holds a credential. `console.host` and `cluster.features.ha.vip` are this side's VIP; the platform domain goes only into `console.globalHost`: + +```json +{ + "basic": { + "username": "admin@cpaas.io", + "password": "${PLATFORM_ADMIN_PASSWORD_B64}" + }, + "registry": { + "domain": "${REGISTRY_DOMAIN}", + "username": "${REGISTRY_USERNAME}", + "password": "${REGISTRY_PASSWORD_B64}" + }, + "console": { + "host": ["${CONTROL_PLANE_VIP}"], + "globalHost": "${PLATFORM_HOST}", + "httpPort": 80, + "httpsPort": 443, + "cert": { + "selfSigned": {} + } + }, + "cluster": { + "clusterCIDR": "${CLUSTER_CIDR}", + "serviceCIDR": "${SERVICE_CIDR}", + "features": { + "ha": { + "vip": "${CONTROL_PLANE_VIP}", + "vport": 6443, + "isThirdParty": true + } + } + }, + "product": ["base", "acp"], + "deployMode": "normal", + "hostIP": "${HOST_IP}" +} +``` + +For the DR pair, prefer one trusted third-party platform certificate on both sides, with `PLATFORM_HOST` in its SAN list, over two unrelated self-signed ones, so that clients see the same certificate whichever side the domain points at. Replace `console.cert` with the `thirdParty` block that Step 8 of the product page describes; the page does not list its field names, so get them from Alauda support, and pass the private key at trigger time like the passwords (a placeholder in the file, added to the second `envsubst` list). This certificate is unrelated to the etcd SANs in `12-kcp.yaml`. + +#### Check before applying + +Render the control plane and the `infra` pool, and check them before anything reaches DCS: + +```bash +# on the bootstrap cluster, in the delivery directory +load_env global-primary.env +mkdir -p rendered +for f in global-primary/1?-*.yaml; do render global-primary.env "$f"; echo '---'; done > rendered/global-primary-cp.yaml +for f in global-primary/2?-infra-*.yaml; do render global-primary.env "$f"; echo '---'; done > rendered/global-primary-infra.yaml +render global-primary.env global-primary/30-import-extra-resources.yaml > rendered/global-primary-30.yaml + +grep -n '\${' rendered/global-primary-*.yaml # expect no output +grep -niE 'authkey|authuser|password' rendered/global-primary-*.yaml # expect no output +grep -c 'provider-id: PROVIDER_ID' rendered/global-primary-cp.yaml # expect 2 +grep -c 'node-ip: NODE_IP' rendered/global-primary-cp.yaml rendered/global-primary-infra.yaml # expect 1 each +grep -n "ip: \"${CONTROL_PLANE_VIP}\"" rendered/global-primary-*.yaml # expect no output: the VIP is in no pool +python3 - "${CLUSTER_CIDR}" "${SERVICE_CIDR}" "${KUBE_OVN_JOIN_CIDR}" "${CP_NODE_IP_1}/${DCS_NODE_MASK}" <<'EOF' +import ipaddress, itertools, sys +nets = [ipaddress.ip_network(a, strict=False) for a in sys.argv[1:]] +bad = [(str(a), str(b)) for a, b in itertools.combinations(nets, 2) if a.overlaps(b)] +print("overlap: %s" % bad if bad else "no overlap") +EOF +python3 -c 'import ipaddress, sys; ipaddress.IPv4Address(sys.argv[1]); print("endpoint is an IPv4 address")' "${CONTROL_PLANE_VIP}" +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get secret "${PROVIDER_SECRET_NAME}" "${DCS_ENCRYPTION_PROVIDER_SECRET}" -o name +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" apply --dry-run=server -f rendered/global-primary-cp.yaml +``` + +Pass: every check prints its expected result, both Secrets exist, and the server-side dry run accepts every resource. Also confirm that the three CIDRs overlap none of the VIP, DCS, other-cluster, or customer networks — the script checks only the node subnet. + +#### Apply the control plane + +Steps 5 and 6 of the product page. The control plane goes first and alone, so a mistake costs three VMs, not six: + +```bash +# on the bootstrap cluster +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" apply -f rendered/global-primary-cp.yaml +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get dcsmachine,machines.cluster.x-k8s.io -o wide # watch the VMs being created +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system wait --for=condition=Ready kubeadmcontrolplane/global-kcp --timeout=60m +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get clusters.cluster.x-k8s.io global -o jsonpath='{.status.phase}{"\n"}' +curl -k "https://${CONTROL_PLANE_VIP}:6443/readyz"; echo +umask 077 +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get secret global-kubeconfig -o jsonpath='{.data.value}' | base64 -d > "${CLUSTER_KUBECONFIG}" +``` + +Pass: the `KubeadmControlPlane` is `Ready`, the `Cluster` phase is `Provisioned`, and the VIP answers `ok`. The last command saves the admin kubeconfig that Cluster API generated for the new cluster, as the common [Kubeconfig convention](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#kubeconfig-convention) describes; it is a credential and stays out of every archive. + +If a `DCSMachine` stays in `Provisioning`, stop and diagnose before you apply anything else, using [Troubleshooting: VM stuck in creating](https://docs.alauda.io/immutable-infra/1.0/infrastructure/huawei-dcs.html#stuck-creating). Deleting and re-applying only hides the cause. + +#### Apply the infra pool + +```bash +# on the bootstrap cluster +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" apply --dry-run=server -f rendered/global-primary-infra.yaml && kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" apply -f rendered/global-primary-infra.yaml +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get machines.cluster.x-k8s.io -l cluster.x-k8s.io/deployment-name=global-infra -o wide +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system wait --for=jsonpath='{.status.readyReplicas}'=3 machinedeployment/global-infra --timeout=60m +``` + +Pass: three `infra` machines are `Running`, and the MachineDeployment reports three ready replicas. + +#### Import the provider Secrets + +```bash +# on the bootstrap cluster +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" apply -f rendered/global-primary-30.yaml +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get cm dcs-import-extra-resources -o jsonpath='{.data.resources\.yaml}' +kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" -n cpaas-system get dcscluster global -o jsonpath='{.spec.credentialSecretRef.name}{" "}{.spec.encryptionProviderConfigRef.name}{"\n"}' +``` + +Pass: both `names` entries are non-empty and equal the two Secret names that the `DCSCluster` references. + +#### Trigger the platform installation + +Step 8 of the product page. The registry user and the passwords are read at the prompt, the passwords base64-encoded as the installer expects, substituted into the rendered request, and streamed to the installer; they never reach a file or a command line. Use the same values on both `global` sides: + +```bash +# on the bootstrap node, in the delivery directory +INSTALLER_IP=$(kubectl --kubeconfig "${BOOTSTRAP_KUBECONFIG}" get pods -n cpaas-system -l service_name=cpaas-installer -o jsonpath='{.items[0].status.podIP}') +read -rsp 'Platform admin password: ' p; echo +PLATFORM_ADMIN_PASSWORD_B64=$(printf '%s' "$p" | base64 -w0) +read -rp 'Platform registry user: ' REGISTRY_USERNAME +read -rsp 'Platform registry password: ' p; echo +REGISTRY_PASSWORD_B64=$(printf '%s' "$p" | base64 -w0); unset p +export PLATFORM_ADMIN_PASSWORD_B64 REGISTRY_USERNAME REGISTRY_PASSWORD_B64 +payload=$(render global-primary.env global-primary/40-installer-config.json \ + | envsubst '${PLATFORM_ADMIN_PASSWORD_B64} ${REGISTRY_USERNAME} ${REGISTRY_PASSWORD_B64}') +unset PLATFORM_ADMIN_PASSWORD_B64 REGISTRY_USERNAME REGISTRY_PASSWORD_B64 +if printf '%s' "$payload" | grep -q '\${'; then + echo "unreplaced variables" +else + printf '%s' "$payload" | curl -k -X POST "http://${INSTALLER_IP}:8080/cpaas-installer/api/config/dcs" \ + -H 'Content-Type: application/json' -d @- +fi +unset payload +``` + +`40-installer-config.json` is the one manifest with intentional placeholders left after `render`: the three runtime values above. The registry user and password are the ones the common article later prompts for when it uploads packages to the standby. + +#### Monitor the installation + +Step 9 of the product page; a `global` cluster typically takes 30 to 60 minutes: + +```bash +# on the bootstrap node +curl "http://${INSTALLER_IP}:8080/cpaas-installer/api/progress" +tail -f /var/cpaas/data/installer.log +``` + +The phases, signals, and common stalls are tabulated under Step 9 of [Installing the global Cluster](https://docs.alauda.io/immutable-infra/1.0/global/install.html). Pass: the progress API reports `status: Success` and `type: Complete`. + +#### Verify the global cluster + +```bash +# on global +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes -o wide # 6 nodes Ready: 3 control plane, 3 infra +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get clusters.platform.tkestack.io global -o jsonpath='{.status.phase}{"\n"}' +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get clustermodule global +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get pods -n cpaas-system | grep -vE 'Running|Completed' # only the header line +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get secret "${PROVIDER_SECRET_NAME}" "${DCS_ENCRYPTION_PROVIDER_SECRET}" "${DCS_KUBERNETES_FILES_SECRET}" -o name +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=global +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.providerID}{"\t"}{.status.addresses[?(@.type=="InternalIP")].address}{"\n"}{end}' +``` + +Then check the Self-built VIP and the control-plane spread, as in the verification steps of [Configure DCSCluster](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-dcs.html#configure-dcscluster) and [Verify Control Plane HA](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-dcs.html#verify-control-plane-ha): + +```bash +# on global +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get dcscluster global \ + -o jsonpath='{.spec.controlPlaneLoadBalancer.type}{" "}{.spec.controlPlaneEndpoint.host}{":"}{.spec.controlPlaneEndpoint.port}{"\n"}' +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get moduleinfo -l cpaas.io/cluster-name=global,cpaas.io/module-name=alive +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get apprelease alive +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n kube-system get pods -l app=alive -o wide +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get --raw=/version +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get dcscluster global \ + -o jsonpath='{range .status.conditions[?(@.type=="ControlPlaneHAReady")]}{.status}{" "}{.reason}{"\n"}{end}' +``` + +Pass: the installation reports `Success`; six nodes are `Ready`, each with a non-empty `providerID` and the IP of its pool slot; both Secrets were imported, and the provider-shipped files Secret exists, because `12-kcp.yaml` and `22-infra-kct.yaml` read it for every new machine; the endpoint type is `internal` with this side's VIP; an `alive` Pod runs on every control-plane node and `/version` answers through the VIP; `ControlPlaneHAReady` is `True` (when `DCS_CONTROL_PLANE_HA` is `true`). + +The `infra` taint is already on the Nodes — taints are applied at registration — but the role labels reach the Nodes only after [Enable label synchronization](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#enable-label-synchronization), the first step of the platform plugins in the common article. Until then, `kubectl get nodes -l node-role.kubernetes.io/infra` returns nothing on this cluster; that is expected, not a wrong label. + +Keep the bootstrap node until [Add disaster recovery](#add-disaster-recovery) is verified. Never delete the `Cluster`, `KubeadmControlPlane`, or DCS objects to clean up: they own the running `global` machines. + +### Add disaster recovery + +The `global` pair is two complete installations, each from its own bootstrap node, that differ only in their cluster env file. Disaster recovery adds three increments that must be **identical on both sides** — the encryption provider Secret built from one shared file, the same etcd server certificate SANs, and the encryption Secret in the import ConfigMap — then an install order driven by the platform domain, and `etcd-sync` on the standby. The authoritative procedure is [Optional Disaster Recovery Deployment](https://docs.alauda.io/immutable-infra/1.0/global/install.html#optional-disaster-recovery-deployment); the DCS notes are in [Provider Notes](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#provider-notes). + +#### Before you start + +- The [network requirements](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#network-requirements) are open in both directions between the two `global` networks: `443`, `6443`, `11443`, `2379`, and `80` only for HTTP access. The direction of traffic reverses after a failover. The product page asks for load-balancer listeners on each VIP; for a Self-built VIP it does not say which of these ports the VIP serves. Test each port from each side against the other side's VIP once both sides are up, and resolve any failure — `2379` in particular, which `etcd-sync` reads through the primary's VIP — before you install `etcd-sync`. +- `PLATFORM_HOST` is a DNS name whose record you can change. Each cluster is always reached through its own VIP; only the platform domain moves. +- Two bootstrap nodes, both prepared as in [Prepare DCS](#prepare-dcs), each holding the same delivery directory, including `global-standby/` created with `cp -r global-primary global-standby`. +- The Bare Metal-only DR steps (shared ServiceAccount signing key, rule ConfigMap) do not apply to DCS; adding them changes objects that the DCS provider does not expect. + +#### Prepare the shared DR inputs + +Generate the encryption key once, on the primary bootstrap node. It is a credential: it decrypts every Secret in the `global` etcd. + +```bash +# on the primary bootstrap node, in the delivery directory +load_env global-primary.env +umask 077 +mkdir -p "$(dirname "${ENCRYPTION_PROVIDER_CONF}")" +ENCRYPTION_PROVIDER_SECRET_B64=$(head -c 32 /dev/urandom | base64) +cat > "${ENCRYPTION_PROVIDER_CONF}" </etcd-sync.amd64..tgz` on the package server); it is not part of the installer bundle and reaches the standby through the common upload step. Follow [Install etcd-sync (ACP 4.4.0+)](https://docs.alauda.io/immutable-infra/1.0/global/install.html#install-etcd-sync): + +```bash +# on global (standby): the plugin must be published here, not only on the primary +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" get moduleplugins | grep -i etcd +``` + +Create the token Secret on the standby. The token is the bearer token of the primary's `cpaas-system/k8sadmin` ServiceAccount; it passes from one cluster to the other through a shell variable and process substitution, and is never displayed, stored, or put on a command line: + +```bash +# on global (primary, then standby) +ACTIVE_CLUSTER_TOKEN=$(kubectl --kubeconfig "${PRIMARY_GLOBAL_KUBECONFIG}" -n cpaas-system get secret k8sadmin -o jsonpath='{.data.token}' | base64 -d) +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system create secret generic etcd-sync-active-cluster-token \ + --from-file=token=<(printf '%s' "${ACTIVE_CLUSTER_TOKEN}") +unset ACTIVE_CLUSTER_TOKEN +``` + +Install the plugin from the standby's web console, opened through `https://${STANDBY_CLUSTER_VIP}`: **Administrator** view, **Marketplace > Cluster Plugins**, cluster `global`, **Alauda Container Platform etcd Synchronizer**, **Install**: + +| Parameter | Value | +| --- | --- | +| Active Global Cluster VIP | The value of `PRIMARY_CLUSTER_VIP` | +| Active Global Cluster ETCD Endpoints | `https://${PRIMARY_CLUSTER_VIP}:2379` | +| Standby Cluster ETCD Endpoints | Default | +| Active Global Cluster Token Secret | `etcd-sync-active-cluster-token` | +| Data Check Interval | Default | +| Print detail logs | Disabled | + +The `etcd-sync-bootstrap` Job runs before the `etcd-sync` Deployment and prepares the remote etcd CA, issuer, and client certificate; it reaches the primary's API server on `6443`. + +#### Verify the pair + +On the standby, verify the plugin runtime, following [Enable etcd Synchronization](https://docs.alauda.io/container_platform/4.4/install/global_dr.html#etcd_sync): + +```bash +# on global (standby) +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n kube-public get configmap etcd-sync -o name # present only on the standby +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system get job etcd-sync-bootstrap +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system get secret remote-etcd-ca remote-etcd-client +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system get issuer remote-etcd-issuer +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system get certificate remote-etcd-client +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system get po -l app=etcd-sync +leader_pod=$(kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system get lease etcd-sync-mirror -o jsonpath='{.spec.holderIdentity}') +kubectl --kubeconfig "${STANDBY_GLOBAL_KUBECONFIG}" -n cpaas-system logs "$leader_pod" | grep -E "Acquired leader lease|Start Sync update" +``` + +The monitor Service only has a ClusterIP, so run the consistency check on a standby control-plane node (log in as `boot`). A check answered with HTTP `425` is not allowed yet; retry until it runs: + +```bash +# on a node: a control-plane node of the standby +mirror_svc=$(sudo kubectl -n cpaas-system get svc etcd-sync-monitor -o jsonpath='{.spec.clusterIP}') +if [[ $mirror_svc =~ ^[0-9a-fA-F:]+$ ]]; then mirror_host="[$mirror_svc]"; else mirror_host="$mirror_svc"; fi +until code=$(curl -sg -o /tmp/etcd-sync-check.txt -w '%{http_code}' "http://$mirror_host/check") && [ "$code" != 425 ]; do sleep 30; done +cat /tmp/etcd-sync-check.txt +``` + +Pass: zero missed and zero surplus keys. Do not accept the pair on the AppRelease status alone. + +Prove that the three increments are identical on both final clusters. The hash compares the encryption key without printing it: + +```bash +# on global (both sides); each pair of lines must be identical +for kc in "${PRIMARY_GLOBAL_KUBECONFIG}" "${STANDBY_GLOBAL_KUBECONFIG}"; do + kubectl --kubeconfig "$kc" -n cpaas-system get secret "${DCS_ENCRYPTION_PROVIDER_SECRET}" -o jsonpath='{.data.encryption-provider\.conf}' | sha256sum +done +for kc in "${PRIMARY_GLOBAL_KUBECONFIG}" "${STANDBY_GLOBAL_KUBECONFIG}"; do + kubectl --kubeconfig "$kc" -n cpaas-system get kubeadmcontrolplane global-kcp \ + -o jsonpath='{.spec.kubeadmConfigSpec.clusterConfiguration.etcd.local.serverCertSANs}{"\n"}' +done +for kc in "${PRIMARY_GLOBAL_KUBECONFIG}" "${STANDBY_GLOBAL_KUBECONFIG}"; do + kubectl --kubeconfig "$kc" -n cpaas-system get dcscluster global -o jsonpath='{.spec.encryptionProviderConfigRef.name}{"\n"}' + kubectl --kubeconfig "$kc" -n cpaas-system get secret "${PROVIDER_SECRET_NAME}" -o name +done +``` + +Then restart the Pods that load the DR and endpoint configuration, on a control-plane node of each side: + +```bash +# on a node: a control-plane node of the primary, then of the standby +sudo kubectl delete po -n cpaas-system -l 'service_name in (alertmanager,vmselect,vminsert)' +sudo kubectl delete po -n cpaas-system -l service_name=cluster-transformer +``` + +Finally, decommission each bootstrap node once its side's Secrets were verified on the final `global` (stage 12 of the common [Delivery flow](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#delivery-flow)): remove only the `minialauda` bootstrap cluster, as described in [Decommission the Bootstrap Cluster](https://docs.alauda.io/immutable-infra/1.0/global/install.html#decommission-bootstrap-cluster), and delete the encryption file from the node. + +The pair is accepted only after a failover drill with [the failover sequence](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#failover-procedure), run at acceptance once the workload clusters exist, so that the drill exercises them. After the switch, verify on the new active side that every workload `Cluster`, `Machine`, and `DCSMachine` is intact, that no VM was deleted, and that both DCS Secrets exist there. + +### Create a workload cluster + +A workload cluster is created by applying its Cluster API resources to `global` — always to the active side, never to the standby. The resources follow the "Using YAML" section of [Creating Clusters on Huawei DCS](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-dcs.html) and [Worker Node Deployment](https://docs.alauda.io/immutable-infra/1.0/manage-nodes/huawei-dcs.html#worker-node-deployment). All commands of this section run on the delivery host, in the delivery directory, after `load_env wl-prod.env`: `GLOBAL_KUBECONFIG` in `wl-prod.env` points at the active `global` and `CLUSTER_KUBECONFIG` at the new cluster. + +#### Check global + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n kube-public get configmap etcd-sync -o name 2>/dev/null || echo "not standby" # must print "not standby" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get pods -n cpaas-system | grep -E 'kubeadm|dcs' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get crd | grep -E '^dcs.*\.infrastructure\.cluster\.x-k8s\.io' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get deploy capi-controller-manager -o jsonpath='{.spec.template.spec.containers[0].args}{"\n"}' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get moduleconfigs -l cpaas.io/module-name=alive +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get secret "${PROVIDER_SECRET_NAME}" "${DCS_KUBERNETES_FILES_SECRET}" -o name +``` + +Pass: this is not the standby; the Kubeadm and DCS provider Pods run and the four DCS CRDs exist (otherwise install the providers as in [Huawei DCS Provider Installation](https://docs.alauda.io/immutable-infra/1.0/install/huawei-dcs.html)); the controller arguments contain `--additional-sync-machine-labels`; both Secrets exist. The `alive` line is an item to validate, not a documented DCS gate: the provider installs the Self-built VIP runtime through an `alive` ModuleInfo, but the DCS pages do not say that `alive` must be published on `global` (the vSphere page does for its provider). If it lists no version, confirm with Alauda support before you create a cluster with `type: internal`. Check the [network paths](#open-the-network-paths) from `global` to DCS and to the new cluster's VIP and nodes as well. + +`CLUSTER_NAME` must not be `global` and must not start with `global-`, or a failover can select this cluster's resources as if they belonged to the `global` pair. + +#### Read the shared values from global + +A workload cluster does not collect its versions again: they already live on `global`, at the places the product's [Resolving Placeholder Values](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-dcs.html#resolving-placeholders) names: + +| Value | Lives on `global` at | Parameter | +| --- | --- | --- | +| Kubernetes version | `data.kubernetesVersion` of the `cpaas.io/dcs-vm-template` ConfigMap | `K8S_VERSION` | +| CoreDNS and etcd tags | `data.corednsTag` and `data.etcdTag` of the same ConfigMap | `DNS_IMAGE_TAG`, `ETCD_IMAGE_TAG` | +| VM template name | Value of the ConfigMap's `cpaas.io/dcs-vm-template` label | `DCS_VM_TEMPLATE` | +| Kube-OVN chart version | `cpaas.io/kube-ovn-version` annotation of `Cluster/global` | `KUBE_OVN_VERSION` | +| Registry address | `cpaas.io/registry-address` annotation of `Cluster/global` | `NODE_REGISTRY_ADDRESS` | +| DCS credential | The Secret imported during the `global` installation | `PROVIDER_SECRET_NAME` | + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get cm -l cpaas.io/dcs-vm-template -o yaml +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get clusters.cluster.x-k8s.io global \ + -o jsonpath='{.metadata.annotations.cpaas\.io/kube-ovn-version}{"\n"}{.metadata.annotations.cpaas\.io/registry-address}{"\n"}' +``` + +Pass: the values match `site.env`, and `NODE_REGISTRY_ADDRESS` in `wl-prod.env` is the printed registry address. If the ConfigMap query returns nothing, apply the ConfigMap shipped with the Alauda OS release first (its format is shown in [Machine Templates](https://docs.alauda.io/immutable-infra/1.0/infrastructure/huawei-dcs.html#machine-templates)). If the cluster runs in another DCS site or under another DCS account, create a separate credential Secret on `global`, named after the cluster (never with the `global-` prefix), with the commands of [Create the DCS credential Secret](#create-the-dcs-credential-secret) run against `GLOBAL_KUBECONFIG` and without the `cpaas.io/cluster-name=global` label, and set `PROVIDER_SECRET_NAME` to its name in the cluster file. The DR pages do not say how such a Secret reaches the standby `global`; confirm that with Alauda support before you rely on it after a failover. + +What a new workload cluster really plans is its name, VIP and VRID, CIDRs, node IPs, sizes, and — in another site — its DCS placement. + +#### Write the manifests + +`wl-prod/10-cp-pool.yaml`: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: DCSIpHostnamePool +metadata: + name: ${CLUSTER_NAME}-cp-pool + namespace: cpaas-system +spec: + pool: + - ip: "${CP_NODE_IP_1}" + mask: "${DCS_NODE_MASK}" + gateway: "${DCS_NODE_GATEWAY}" + dns: "${DCS_NODE_DNS}" + hostname: "${NODE_NAME_PREFIX}-cp-1" + machineName: "${NODE_NAME_PREFIX}-cp-1" + persistentDisk: + - slot: 0 + quantityGB: ${DCS_CP_CPAAS_DISK_GB} + datastoreName: "${DCS_DATASTORE}" + path: /var/cpaas + format: xfs + mountOptions: + - defaults + - ip: "${CP_NODE_IP_2}" + mask: "${DCS_NODE_MASK}" + gateway: "${DCS_NODE_GATEWAY}" + dns: "${DCS_NODE_DNS}" + hostname: "${NODE_NAME_PREFIX}-cp-2" + machineName: "${NODE_NAME_PREFIX}-cp-2" + persistentDisk: + - slot: 0 + quantityGB: ${DCS_CP_CPAAS_DISK_GB} + datastoreName: "${DCS_DATASTORE}" + path: /var/cpaas + format: xfs + mountOptions: + - defaults + - ip: "${CP_NODE_IP_3}" + mask: "${DCS_NODE_MASK}" + gateway: "${DCS_NODE_GATEWAY}" + dns: "${DCS_NODE_DNS}" + hostname: "${NODE_NAME_PREFIX}-cp-3" + machineName: "${NODE_NAME_PREFIX}-cp-3" + persistentDisk: + - slot: 0 + quantityGB: ${DCS_CP_CPAAS_DISK_GB} + datastoreName: "${DCS_DATASTORE}" + path: /var/cpaas + format: xfs + mountOptions: + - defaults +``` + +`wl-prod/11-cp-template.yaml`: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: DCSMachineTemplate +metadata: + name: ${CLUSTER_NAME}-cp-template + namespace: cpaas-system +spec: + template: + spec: + vmTemplateName: "${DCS_VM_TEMPLATE}" + location: + type: folder + name: "${DCS_VM_FOLDER}" + vmConfig: + dvSwitchName: "${DCS_DVSWITCH}" + portGroupName: "${DCS_PORT_GROUP}" + dcsMachineCpuSpec: + quantity: ${DCS_CP_CPU} + dcsMachineMemorySpec: # MB + quantity: ${DCS_CP_MEMORY_MB} + dcsMachineDiskSpec: # GB + - quantity: 0 + datastoreName: "${DCS_DATASTORE}" + systemVolume: true + - quantity: 10 + datastoreName: "${DCS_DATASTORE}" + path: /var/lib/etcd + format: xfs + - quantity: 100 + datastoreName: "${DCS_DATASTORE}" + path: /var/lib/kubelet + format: xfs + - quantity: 100 + datastoreName: "${DCS_DATASTORE}" + path: /var/lib/containerd + format: xfs + ipHostPoolRef: + name: ${CLUSTER_NAME}-cp-pool +``` + +`wl-prod/12-kcp.yaml` — a workload control plane skips both CoreDNS and kube-proxy, which the platform deploys, and carries no etcd SANs: + +```yaml +apiVersion: controlplane.cluster.x-k8s.io/v1beta1 +kind: KubeadmControlPlane +metadata: + name: ${CLUSTER_NAME}-kcp + namespace: cpaas-system + annotations: + controlplane.cluster.x-k8s.io/skip-coredns: "" + controlplane.cluster.x-k8s.io/skip-kube-proxy: "" +spec: + replicas: 3 + version: ${K8S_VERSION} + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + machineTemplate: + nodeDrainTimeout: 1m + nodeDeletionTimeout: 5m + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: DCSMachineTemplate + name: ${CLUSTER_NAME}-cp-template + kubeadmConfigSpec: + format: ignition + users: + - name: boot + sshAuthorizedKeys: + - "${BOOT_SSH_PUBLIC_KEY}" + files: + - path: /etc/kubernetes/admission/psa-config.yaml + owner: "root:root" + permissions: "0644" + contentFrom: + secret: + name: ${DCS_KUBERNETES_FILES_SECRET} + key: psa-config.yaml + - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json + owner: "root:root" + permissions: "0644" + contentFrom: + secret: + name: ${DCS_KUBERNETES_FILES_SECRET} + key: control-plane-kubelet-patch.json + - path: /etc/kubernetes/audit/policy.yaml + owner: "root:root" + permissions: "0644" + contentFrom: + secret: + name: ${DCS_KUBERNETES_FILES_SECRET} + key: audit-policy.yaml + preKubeadmCommands: + - while ! ip route | grep -q "default via"; do sleep 1; done; echo "NetworkManager started" + - mkdir -p /run/cluster-api && restorecon -Rv /run/cluster-api + - if [ -f /etc/disk-setup.sh ]; then bash /etc/disk-setup.sh; fi + postKubeadmCommands: + - chmod 600 /var/lib/kubelet/config.yaml + clusterConfiguration: + imageRepository: ${IMAGE_REPOSITORY} + dns: + imageTag: "${DNS_IMAGE_TAG}" + etcd: + local: + imageTag: "${ETCD_IMAGE_TAG}" + apiServer: + extraArgs: + audit-log-format: json + audit-log-maxage: "30" + audit-log-maxbackup: "10" + audit-log-maxsize: "200" + profiling: "false" + audit-log-mode: batch + audit-log-path: /etc/kubernetes/audit/audit.log + audit-policy-file: /etc/kubernetes/audit/policy.yaml + tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384" + encryption-provider-config: /etc/kubernetes/encryption-provider.conf + admission-control-config-file: /etc/kubernetes/admission/psa-config.yaml + tls-min-version: VersionTLS12 + kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt + extraVolumes: + - name: vol-dir-0 + hostPath: /etc/kubernetes + mountPath: /etc/kubernetes + pathType: Directory + controllerManager: + extraArgs: + bind-address: "::" + profiling: "false" + tls-min-version: VersionTLS12 + flex-volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" + scheduler: + extraArgs: + bind-address: "::" + tls-min-version: VersionTLS12 + profiling: "false" + initConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + node-labels: "kube-ovn/role=master" + provider-id: PROVIDER_ID + volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" + protect-kernel-defaults: "true" + joinConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + node-ip: NODE_IP + node-labels: "kube-ovn/role=master" + provider-id: PROVIDER_ID + volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" + protect-kernel-defaults: "true" +``` + +`wl-prod/13-dcscluster.yaml`: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: DCSCluster +metadata: + name: ${CLUSTER_NAME} + namespace: cpaas-system +spec: + controlPlaneLoadBalancer: + host: "${CONTROL_PLANE_VIP}" + port: 6443 + type: internal + vrid: ${DCS_VRID} + controlPlaneEndpoint: + host: "${CONTROL_PLANE_VIP}" + port: 6443 + credentialSecretRef: + name: "${PROVIDER_SECRET_NAME}" + controlPlaneHA: + enabled: ${DCS_CONTROL_PLANE_HA} + networkType: kube-ovn + site: "${DCS_SITE}" +``` + +`wl-prod/14-cluster.yaml` — the annotations the operator authors on a workload `Cluster`; ACP writes the computed ones (`cpaas.io/nodes-number` and the like) itself. `cpaas.io/os-family` is omitted on purpose, because it is only for KubeOS templates: + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: ${CLUSTER_NAME} + namespace: cpaas-system + labels: + cluster-type: DCS + annotations: + capi.cpaas.io/resource-group-version: infrastructure.cluster.x-k8s.io/v1beta1 + capi.cpaas.io/resource-kind: DCSCluster + capi.cpaas.io/kubernetes: "${K8S_VERSION}" + cpaas.io/kube-ovn-join-cidr: "${KUBE_OVN_JOIN_CIDR}" + cpaas.io/kube-ovn-version: "${KUBE_OVN_VERSION}" + cpaas.io/registry-address: "${NODE_REGISTRY_ADDRESS}" + cpaas.io/nodes-mode: self-managed +spec: + clusterNetwork: + pods: + cidrBlocks: + - "${CLUSTER_CIDR}" + services: + cidrBlocks: + - "${SERVICE_CIDR}" + controlPlaneRef: + apiVersion: controlplane.cluster.x-k8s.io/v1beta1 + kind: KubeadmControlPlane + name: ${CLUSTER_NAME}-kcp + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: DCSCluster + name: ${CLUSTER_NAME} +``` + +The workload cluster inherits the registry of `global`. To bind it to a dedicated registry instead, add the `cpaas.io/registry-reference` label before the first apply — it is read only at creation; see [Choose the Image Registry for a Workload Cluster](https://docs.alauda.io/immutable-infra/1.0/how-to/workload-cluster-image-registry.html). + +`wl-prod/20-infra-pool.yaml`: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: DCSIpHostnamePool +metadata: + name: ${CLUSTER_NAME}-infra-pool + namespace: cpaas-system +spec: + pool: + - ip: "${INFRA_NODE_IP_1}" + mask: "${DCS_NODE_MASK}" + gateway: "${DCS_NODE_GATEWAY}" + dns: "${DCS_NODE_DNS}" + hostname: "${NODE_NAME_PREFIX}-infra-1" + machineName: "${NODE_NAME_PREFIX}-infra-1" + persistentDisk: + - slot: 0 + quantityGB: ${DCS_INFRA_CPAAS_DISK_GB} + datastoreName: "${DCS_DATASTORE}" + path: /var/cpaas + format: xfs + mountOptions: + - defaults + - ip: "${INFRA_NODE_IP_2}" + mask: "${DCS_NODE_MASK}" + gateway: "${DCS_NODE_GATEWAY}" + dns: "${DCS_NODE_DNS}" + hostname: "${NODE_NAME_PREFIX}-infra-2" + machineName: "${NODE_NAME_PREFIX}-infra-2" + persistentDisk: + - slot: 0 + quantityGB: ${DCS_INFRA_CPAAS_DISK_GB} + datastoreName: "${DCS_DATASTORE}" + path: /var/cpaas + format: xfs + mountOptions: + - defaults + - ip: "${INFRA_NODE_IP_3}" + mask: "${DCS_NODE_MASK}" + gateway: "${DCS_NODE_GATEWAY}" + dns: "${DCS_NODE_DNS}" + hostname: "${NODE_NAME_PREFIX}-infra-3" + machineName: "${NODE_NAME_PREFIX}-infra-3" + persistentDisk: + - slot: 0 + quantityGB: ${DCS_INFRA_CPAAS_DISK_GB} + datastoreName: "${DCS_DATASTORE}" + path: /var/cpaas + format: xfs + mountOptions: + - defaults +``` + +`wl-prod/21-infra-template.yaml`: + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: DCSMachineTemplate +metadata: + name: ${CLUSTER_NAME}-infra-template + namespace: cpaas-system +spec: + template: + spec: + vmTemplateName: "${DCS_VM_TEMPLATE}" + location: + type: folder + name: "${DCS_VM_FOLDER}" + vmConfig: + dvSwitchName: "${DCS_DVSWITCH}" + portGroupName: "${DCS_PORT_GROUP}" + dcsMachineCpuSpec: + quantity: ${DCS_INFRA_CPU} + dcsMachineMemorySpec: # MB + quantity: ${DCS_INFRA_MEMORY_MB} + dcsMachineDiskSpec: # GB + - quantity: 0 + datastoreName: "${DCS_DATASTORE}" + systemVolume: true + - quantity: 100 + datastoreName: "${DCS_DATASTORE}" + path: /var/lib/kubelet + format: xfs + - quantity: 100 + datastoreName: "${DCS_DATASTORE}" + path: /var/lib/containerd + format: xfs + ipHostPoolRef: + name: ${CLUSTER_NAME}-infra-pool +``` + +`wl-prod/22-infra-kct.yaml`: + +```yaml +apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 +kind: KubeadmConfigTemplate +metadata: + name: ${CLUSTER_NAME}-infra-kct + namespace: cpaas-system +spec: + template: + spec: + format: ignition + users: + - name: boot + sshAuthorizedKeys: + - "${BOOT_SSH_PUBLIC_KEY}" + files: + - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json + owner: "root:root" + permissions: "0644" + contentFrom: + secret: + name: ${DCS_KUBERNETES_FILES_SECRET} + key: worker-kubelet-patch.json + preKubeadmCommands: + - while ! ip route | grep -q "default via"; do sleep 1; done; echo "NetworkManager started" + - mkdir -p /run/cluster-api && restorecon -Rv /run/cluster-api + - if [ -f /etc/disk-setup.sh ]; then bash /etc/disk-setup.sh; fi + postKubeadmCommands: + - chmod 600 /var/lib/kubelet/config.yaml + joinConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + node-ip: NODE_IP + provider-id: PROVIDER_ID + volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" + protect-kernel-defaults: "true" +``` + +`wl-prod/23-infra-md.yaml`: + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: MachineDeployment +metadata: + name: ${CLUSTER_NAME}-infra + namespace: cpaas-system +spec: + clusterName: ${CLUSTER_NAME} + replicas: 3 + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + maxUnavailable: 1 + selector: + matchLabels: null + template: + metadata: + labels: + cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME} + pool.name: ${CLUSTER_NAME}-infra + node-role.kubernetes.io/infra: "" + node-role.kubernetes.io/worker: "" + spec: + clusterName: ${CLUSTER_NAME} + version: ${K8S_VERSION} + nodeDrainTimeout: 1m + nodeDeletionTimeout: 5m + taints: + - key: node-role.kubernetes.io/infra + value: reserved + effect: NoSchedule + bootstrap: + configRef: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: ${CLUSTER_NAME}-infra-kct + namespace: cpaas-system + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: DCSMachineTemplate + name: ${CLUSTER_NAME}-infra-template + namespace: cpaas-system +``` + +A second workload cluster is `cp -r wl-prod wl-prod2` plus its own `wl-prod2.env`. The manifests stay untouched, which is what keeps one cluster's values out of another. + +#### Apply the control plane + +Render and check both groups, then apply the control plane and verify it before any pool exists: + +```bash +# on global, in the delivery directory +for f in wl-prod/1?-*.yaml; do render wl-prod.env "$f"; echo '---'; done > rendered/wl-prod-cp.yaml +for f in wl-prod/2?-infra-*.yaml; do render wl-prod.env "$f"; echo '---'; done > rendered/wl-prod-infra.yaml +grep -n '\${' rendered/wl-prod-*.yaml # expect no output +grep -nE '^ name: "?global|clusterName: global|cluster-name: "?global' rendered/wl-prod-*.yaml # expect no output +grep -c 'provider-id: PROVIDER_ID' rendered/wl-prod-cp.yaml # expect 2 +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" apply --dry-run=server -f rendered/wl-prod-cp.yaml +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" apply -f rendered/wl-prod-cp.yaml +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system wait --for=condition=Ready kubeadmcontrolplane/${CLUSTER_NAME}-kcp --timeout=60m +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get clusters.cluster.x-k8s.io ${CLUSTER_NAME} +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get machines.cluster.x-k8s.io,dcsmachine -l cluster.x-k8s.io/cluster-name=${CLUSTER_NAME} -o wide +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get clustermodule ${CLUSTER_NAME} -o jsonpath='{.status.base.deployStatus}{"\n"}' +``` + +Run the CIDR and endpoint checks of [Check before applying](#check-before-applying) with this cluster's values as well; the workload CIDRs must also stay clear of the `global` CIDRs. Then verify the Self-built VIP and the spread, and save the kubeconfig: + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get moduleinfo -l cpaas.io/cluster-name=${CLUSTER_NAME},cpaas.io/module-name=alive +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get dcscluster ${CLUSTER_NAME} \ + -o jsonpath='{range .status.conditions[?(@.type=="ControlPlaneHAReady")]}{.status}{" "}{.reason}{"\n"}{end}' +umask 077 +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get secret ${CLUSTER_NAME}-kubeconfig -o jsonpath='{.data.value}' | base64 -d > "${CLUSTER_KUBECONFIG}" +``` + +```bash +# on the workload cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get apprelease alive +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n kube-system get pods -l app=alive -o wide +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get --raw=/version +``` + +Pass: the control plane is `Ready` with three machines, the base module reports its deploy status as completed, the `alive` ModuleInfo exists and its Pods run on every control-plane node, `/version` answers through the VIP, and `ControlPlaneHAReady` is `True`. + +#### Apply the infra pool + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" apply --dry-run=server -f rendered/wl-prod-infra.yaml && kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" apply -f rendered/wl-prod-infra.yaml +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system wait --for=jsonpath='{.status.readyReplicas}'=3 machinedeployment/${CLUSTER_NAME}-infra --timeout=60m +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get dcsiphostnamepool ${CLUSTER_NAME}-infra-pool -o jsonpath='{.status.persistentDiskStatus}{"\n"}' +``` + +```bash +# on the workload cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes -o wide +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes -l node-role.kubernetes.io/infra +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.providerID}{"\t"}{.spec.taints}{"\n"}{end}' +``` + +Pass: six nodes are `Ready`, each with a `providerID` and the IP of its pool slot; the three `infra` Nodes — not only their Machines — carry both role labels and the `infra` taint; the pool reports a persistent disk for every slot. + +### Variants + +#### Single cluster + +`global` doubles as the workload cluster: one bootstrap node, no standby, no `etcd-sync`, no workload clusters. Skip [Add disaster recovery](#add-disaster-recovery), set `CONTROL_PLANE_VIP` directly in `global-primary.env` (the DR variables are not needed), set `INGRESS_VIP_RANGE` and `INGRESS_VIP` there as the common [Single-cluster variant](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#single-cluster-variant) describes, and do not create the encryption Secret — the provider then generates the key itself, and encryption at rest stays on. Remove the three increments: + +```diff +# global-primary/12-kcp.yaml +- serverCertSANs: # DR increment: the same list on both sides ++ serverCertSANs: +- - "${PRIMARY_CLUSTER_VIP}" +- - "${STANDBY_CLUSTER_VIP}" +- - "${PLATFORM_HOST}" +- - "etcd.kube-system" ++ - "${CONTROL_PLANE_VIP}" ++ - "${PLATFORM_HOST}" + +# global-primary/13-dcscluster.yaml +- encryptionProviderConfigRef: # DR increment +- name: "${DCS_ENCRYPTION_PROVIDER_SECRET}" + +# global-primary/30-import-extra-resources.yaml + names: + - "${PROVIDER_SECRET_NAME}" +- - "${DCS_ENCRYPTION_PROVIDER_SECRET}" # DR increment +``` + +Drop `"${DCS_ENCRYPTION_PROVIDER_SECRET}"` from the Secret checks in [Check before applying](#check-before-applying) and [Verify the global cluster](#verify-the-global-cluster). The `infra` pool then also carries the application ingress. + +#### Dedicated pools + +For larger workload clusters, replace the `infra` pool with `ingress` (files `30`–`33`), `monitor` (`34`–`37`), and `log` (`40`–`43`) pools. Each is the `infra` pool with its own name, role label, and taint key — distinct keys make a misplaced plugin `Pending` instead of letting it land silently on the wrong pool — and every pool keeps `node-role.kubernetes.io/worker`. Derive the files, then add the new variables (`INGRESS_NODE_IP_1` to `INGRESS_NODE_IP_3`, `DCS_INGRESS_CPU`, `DCS_INGRESS_MEMORY_MB`, `DCS_INGRESS_CPAAS_DISK_GB`, and the same for `MONITOR` and `LOG`) to the cluster file: + +```bash +# on the delivery host, in the delivery directory +for i in 0 1 2 3; do + for f in wl-prod/2$i-infra-*.yaml; do + sed -e 's/-infra/-ingress/g' -e 's#/infra#/ingress#g' -e 's/INFRA_/INGRESS_/g' "$f" > "$(echo "$f" | sed "s#/2$i-infra-#/3$i-ingress-#")" + sed -e 's/-infra/-monitor/g' -e 's#/infra#/monitor#g' -e 's/INFRA_/MONITOR_/g' "$f" > "$(echo "$f" | sed "s#/2$i-infra-#/3$((i + 4))-monitor-#")" + sed -e 's/-infra/-log/g' -e 's#/infra#/log#g' -e 's/INFRA_/LOG_/g' "$f" > "$(echo "$f" | sed "s#/2$i-infra-#/4$i-log-#")" + done +done +rm wl-prod/2?-infra-*.yaml +grep -n 'infra' wl-prod/3*.yaml wl-prod/4*.yaml # expect only infrastructureRef and infrastructure.cluster.x-k8s.io +``` + +The result for `wl-prod/33-ingress-md.yaml`, compared with the `infra` file: + +```diff +- name: ${CLUSTER_NAME}-infra ++ name: ${CLUSTER_NAME}-ingress +... +- pool.name: ${CLUSTER_NAME}-infra +- node-role.kubernetes.io/infra: "" ++ pool.name: ${CLUSTER_NAME}-ingress ++ node-role.kubernetes.io/ingress: "" + node-role.kubernetes.io/worker: "" +... +- - key: node-role.kubernetes.io/infra ++ - key: node-role.kubernetes.io/ingress + value: reserved + effect: NoSchedule +... +- name: ${CLUSTER_NAME}-infra-kct ++ name: ${CLUSTER_NAME}-ingress-kct +... +- name: ${CLUSTER_NAME}-infra-template ++ name: ${CLUSTER_NAME}-ingress-template +``` + +Render and apply each pool on its own, as in [Apply the infra pool](#apply-the-infra-pool), with the file group `wl-prod/3?-ingress-*.yaml`, then `wl-prod/3?-monitor-*.yaml`, then `wl-prod/4?-log-*.yaml`, and check each pool's Nodes before the next. Size the dedicated pools from the product capacity pages; do not scale the consolidated values. + +#### Worker pools + +The main line creates only the control plane and the tainted `infra` pool, so a cluster built from it has no untainted node for applications. Add the customer's worker pools before handing the cluster over; they follow the customer's machines. One generic, untainted pool is the `infra` pool without the `infra` role label and taint — the same four resources as the product's [Worker Node Deployment](https://docs.alauda.io/immutable-infra/1.0/manage-nodes/huawei-dcs.html#worker-node-deployment), with the persistent `/var/cpaas` disk and `maxSurge: 0`. Derive it as files `44`–`47` from the `infra` files — with the dedicated layout, before the `rm` of [Dedicated pools](#dedicated-pools) removes them — then add `WORKER_NODE_IP_1` to `WORKER_NODE_IP_3` (or as many slots as the pool has replicas), `DCS_WORKER_CPU`, `DCS_WORKER_MEMORY_MB`, and `DCS_WORKER_CPAAS_DISK_GB` to the cluster file: + +```bash +# on the delivery host, in the delivery directory (use global-primary/ for a single-cluster global) +for i in 0 1 2 3; do + for f in wl-prod/2$i-infra-*.yaml; do + sed -e 's/-infra/-worker/g' -e 's/INFRA_/WORKER_/g' \ + -e '/node-role.kubernetes.io\/infra: ""/d' -e '/^ taints:$/,/^ effect: NoSchedule$/d' \ + "$f" > "$(echo "$f" | sed "s#/2$i-infra-#/4$((i + 4))-worker-#")" + done +done +grep -n 'infra\|taints' wl-prod/4?-worker-*.yaml # expect only infrastructureRef and infrastructure.cluster.x-k8s.io +``` + +The result for `wl-prod/47-worker-md.yaml`, compared with the `infra` file: + +```diff +- name: ${CLUSTER_NAME}-infra ++ name: ${CLUSTER_NAME}-worker +... +- pool.name: ${CLUSTER_NAME}-infra +- node-role.kubernetes.io/infra: "" ++ pool.name: ${CLUSTER_NAME}-worker + node-role.kubernetes.io/worker: "" +... +- taints: +- - key: node-role.kubernetes.io/infra +- value: reserved +- effect: NoSchedule +... +- name: ${CLUSTER_NAME}-infra-kct ++ name: ${CLUSTER_NAME}-worker-kct +... +- name: ${CLUSTER_NAME}-infra-template ++ name: ${CLUSTER_NAME}-worker-template +``` + +`node-role.kubernetes.io/worker` stays, so the pool receives the machine configuration of the default worker MachineConfigPool. Render and apply the group `wl-prod/4?-worker-*.yaml` as in [Apply the infra pool](#apply-the-infra-pool), with the replica count of the `MachineDeployment` no larger than the number of slots in its pool. + +### Continue with the common steps + +For the `global` pair, the common plugin and machine-configuration steps already ran inside the DR order (steps 3 and 6 of [Install the primary, then the standby](#install-the-primary-then-the-standby)). For each workload cluster, continue in the common article now: + +1. [Install the platform plugins](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#install-the-platform-plugins), placed on the `infra` pool (or on the dedicated pools). +2. [Configure machines](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#configure-machines). +3. Add the customer's worker pools ([Worker pools](#worker-pools)); until then the cluster has no untainted node for applications. +4. [Accept the delivery](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#accept-the-delivery) on every cluster, adding the DCS checks below. + +DCS acceptance checks, for each cluster after `load_env` of its file (for the `global` pair, run them once with `global-primary.env` and once with `global-standby.env`; there `GLOBAL_KUBECONFIG` is the side itself): + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get kubeadmcontrolplane ${CLUSTER_NAME}-kcp -o jsonpath='{.spec.rolloutStrategy.rollingUpdate.maxSurge}{"\n"}' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name=${CLUSTER_NAME} \ + -o jsonpath='{range .items[*]}{.metadata.name}{" maxSurge="}{.spec.strategy.rollingUpdate.maxSurge}{"\n"}{end}' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get dcsmachine -l cluster.x-k8s.io/cluster-name=${CLUSTER_NAME} \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.networkConfig.ip}{"\n"}{end}' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get dcsiphostnamepool -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.persistentDiskStatus}{"\n"}{end}' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get dcscluster ${CLUSTER_NAME} \ + -o jsonpath='{.spec.controlPlaneLoadBalancer.type}{" "}{range .status.conditions[?(@.type=="ControlPlaneHAReady")]}{.status}{end}{"\n"}' +``` + +```bash +# on a node: each node of the cluster, logged in as boot +findmnt /var/cpaas +``` + +Pass: + +- `maxSurge` is `0` on the control plane and on every MachineDeployment. +- Every `DCSMachine` has the IP of its pool slot, and every node has a non-empty `providerID` (the common node check). +- Every pool reports its persistent disks, and `/var/cpaas` is a single dedicated mount on every node, holding the platform state, the monitoring data, and the log collector's working data. Kafka, Keeper, and ClickHouse use StorageClass volumes instead; see [Disks](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#disks). +- The endpoint type is `internal`, and `ControlPlaneHAReady` is `True`. +- Both `global` sides hold the DCS credential Secret and the same encryption Secret (see [Verify the pair](#verify-the-pair)), and the failover drill passed. +- The credential scan of [Accept the delivery](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#accept-the-delivery), `grep -rnIiE 'aescbc|password[[:space:]]*[:=][[:space:]]*[^[:space:]]|_TOKEN=|BEGIN [A-Z ]*PRIVATE KEY' .` after every manifest is rendered into `rendered/`, finds nothing: `40-installer-config.json` holds only its three runtime placeholders, and the kubeconfigs and the encryption file are outside the delivery directory. + +## Related Information + +- [Installing the global Cluster (Immutable Infrastructure)](https://docs.alauda.io/immutable-infra/1.0/global/install.html) +- [Global Cluster Disaster Recovery (Immutable Infrastructure)](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html) +- [Creating Clusters on Huawei DCS](https://docs.alauda.io/immutable-infra/1.0/create-cluster/huawei-dcs.html) +- [Infrastructure Resources for Huawei DCS](https://docs.alauda.io/immutable-infra/1.0/infrastructure/huawei-dcs.html) +- [Managing Nodes on Huawei DCS](https://docs.alauda.io/immutable-infra/1.0/manage-nodes/huawei-dcs.html) +- [Huawei DCS Provider](https://docs.alauda.io/immutable-infra/1.0/overview/providers/huawei-dcs.html) and its [release notes](https://docs.alauda.io/immutable-infra/1.0/release-notes/huawei_dcs.html) +- [Plan the Control Plane Endpoint](https://docs.alauda.io/immutable-infra/1.0/how-to/control-plane-endpoint.html) +- [Configure Annotations, Labels, and Taints on Nodes](https://docs.alauda.io/immutable-infra/1.0/how-to/node-configuration.html) +- [Communication Matrix (ACP 4.4)](https://docs.alauda.io/container_platform/4.4/install/prepare/communication_matrix.html) +- [Global Cluster Disaster Recovery (ACP 4.4)](https://docs.alauda.io/container_platform/4.4/install/global_dr.html) +- [How to Deliver ACP with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md) and the provider articles for [Huawei Cloud Stack](/solutions/acp/How_to_Deliver_ACP_on_Huawei_Cloud_Stack_with_Immutable_Infrastructure.md), [Bare Metal](/solutions/acp/How_to_Deliver_ACP_on_Bare_Metal_with_Immutable_Infrastructure.md), and [VMware vSphere](/solutions/acp/How_to_Deliver_ACP_on_VMware_vSphere_with_Immutable_Infrastructure.md) diff --git a/docs/en/solutions/acp/How_to_Deliver_ACP_on_VMware_vSphere_with_Immutable_Infrastructure.md b/docs/en/solutions/acp/How_to_Deliver_ACP_on_VMware_vSphere_with_Immutable_Infrastructure.md new file mode 100644 index 000000000..5fbef6534 --- /dev/null +++ b/docs/en/solutions/acp/How_to_Deliver_ACP_on_VMware_vSphere_with_Immutable_Infrastructure.md @@ -0,0 +1,1912 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 4.4.x +--- + +# How to Deliver ACP on VMware vSphere with Immutable Infrastructure + +## Purpose + +This guide is the VMware vSphere part of the standard delivery of Alauda Container Platform (ACP) 4.4 on +Immutable Infrastructure (Alauda OS nodes). It covers **workload clusters only**: the vSphere-specific +design decisions, the parameters, preparing vCenter and the `global` cluster, and the complete manifest +set for one workload cluster — three control-plane nodes and the `infra` pool — with the checks that prove +each step worked. + +Scope and assumptions: + +- **`global` is never on vSphere in this delivery.** It already runs on another provider and was delivered + with [Huawei DCS](/solutions/acp/How_to_Deliver_ACP_on_Huawei_DCS_with_Immutable_Infrastructure.md), + [Huawei Cloud Stack](/solutions/acp/How_to_Deliver_ACP_on_Huawei_Cloud_Stack_with_Immutable_Infrastructure.md) + or [Bare Metal](/solutions/acp/How_to_Deliver_ACP_on_Bare_Metal_with_Immutable_Infrastructure.md). + This article therefore has no "Install the global cluster" and no "Add disaster recovery" section; + the `global` cluster and its standby are covered there. +- **Read the common article first:** + [How to Deliver ACP with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md). + The reference design, the parameter files and the `render` helper, the platform plugins, machine + configuration, sizing and acceptance live there and are not repeated here. +- **Versions:** ACP 4.4, VMware vSphere Infrastructure Provider `v1.0.17`, and the Alauda OS + OVA published for the delivered ACP 4.4 release, with the Kubernetes, CoreDNS, etcd and Kube-OVN + versions from that release's row in the + [OS Support Matrix](https://docs.alauda.io/immutable-infra/1.0/overview/os-support-matrix.html#version-mapping) + (Kubernetes 1.35 on ACP 4.4). +- Deliveries on a traditional operating system are not covered here; see the + [standard installation path](https://docs.alauda.io/container_platform/4.4/install/). + +**Validation status.** The manifests in this article are assembled from the ACP 4.4 product +documentation and the VMware vSphere Provider `v1.0.17` documentation. Validate them in a staging +environment before you use them in a production delivery. + +## Resolution + +### VMware vSphere-specific design + +#### Control-plane endpoint: Self-built VIP + +The workload cluster's API endpoint is a **Self-built VIP**: +`VSphereCluster.spec.controlPlaneLoadBalancer.type: internal`. The provider injects the VIP on the first +control-plane node while `kubeadm init` runs, then installs Alive (keepalived and IPVS) into the +workload cluster through a `ModuleInfo` and keeps the VIP on the control-plane nodes. No load balancer +device is involved. + +- **It needs vSphere Provider `v1.0.17` or later and ACP 4.4 or later**, the `alive` cluster plugin + deployable on `global`, VRRP (IP protocol `112`, multicast `224.0.0.18`) and gratuitous ARP allowed + between the control-plane nodes — check the dvPortGroup security policy and any NSX-T distributed + firewall rule — and an `ip_vs`-capable VM template. See + [Self-built VIP prerequisites](https://docs.alauda.io/immutable-infra/1.0/create-cluster/vmware-vsphere.html#self-built-vip-prerequisites). +- **It is a creation-time choice and the field is write-once.** `type`, `host`, `port`, `vrid` and + `interface` are frozen once the cluster exists, and a cluster created without the field — including + one created before the provider was upgraded to `v1.0.17` — can never adopt the VIP. Correcting a + mistake means rebuilding the cluster, so decide and check everything before the first apply. +- **The VIP is an IPv4 address, never a domain name.** The API server certificate is issued by the + cluster's own CA, so a customer certificate for a domain name can never be served on the endpoint. + It must be unused, in the same Layer-2 segment as the control-plane node IPs, not one of the slot + addresses of this cluster, and outside every external IPAM range — the provider checks slot addresses + but cannot see IPAM pools. +- **The VRID is unique on the segment.** Two clusters with the same VRID on one Layer-2 segment take part + in each other's VRRP election. +- `interface` is left out: with one NIC the provider picks the interface that holds the node's primary IP. + With a second NIC, set it explicitly at creation — a wrong auto-detected interface cannot be patched. +- **Alternative: an external Layer 4 load balancer.** Leave `controlPlaneLoadBalancer` out entirely + rather than writing `type: external` — an absent field already means an external load balancer, and + writing it only pins a frozen second copy of the address. See + [External load balancer instead of the Self-built VIP](#external-load-balancer-instead-of-the-self-built-vip). + +#### Node slots: `VSphereMachineConfigPool` + +Every node is cut from a slot in a `VSphereMachineConfigPool`: a fixed hostname, a static IP with gateway +and DNS, and the node's data disks. The rules that keep this working: + +- **One pool per controller.** `-cp-pool` serves the `KubeadmControlPlane`, `-infra-pool` + the `infra` `MachineDeployment`, and each dedicated pool gets its own. The provider records the owning + controller on first allocation and rejects a second one. +- **Hostnames are `--`.** They must be unique across every cluster in the vCenter, not + only within this cluster: the vSphere CPI can fall back to a lookup by guest hostname, and an old VM + that still reports the same hostname makes it reject the new node, which then never gets a + `providerID`. Deriving the hostname from the cluster name rules that out by construction. +- **Plan every slot completely before the first apply.** Once a slot is `InUse` or `Released`, its IP and + disks are frozen; only additional disks are accepted. Hostnames and primary IPs must be unique across + all pools of the cluster — the admission webhook enforces it. +- **Keep pool names for the life of the cluster.** A pool with a new name owns new slots and new VMDKs. + Retrying a failed creation under new pool names accumulates complete disk sets on the datastore until + provider-managed reclaim finishes. Never delete VMDKs or remove pool finalizers by hand. See + [Persistent disk lifecycle](https://docs.alauda.io/immutable-infra/1.0/infrastructure/vmware-vsphere.html#persistent-disk-lifecycle). + +#### Disks + +- **The standard disk set, per slot:** `/var/cpaas` as a persistent disk; `/var/lib/kubelet`, + `/var/lib/containerd` and, on control-plane nodes, `/var/lib/etcd` as ephemeral disks. A persistent disk + is reattached to the VM that replaces the node, so platform state and the monitoring and log data under + `/var/cpaas` survive a node replacement. Ephemeral disks are recreated empty, so a replacement + control-plane node never carries stale etcd member data into `kubeadm join`. +- **The split is a creation-time layout** (`ephemeralDisks[]` exists from provider `v1.0.17`). A slot can + never move a disk from one list to the other later, and sizes of allocated disks cannot change — size + `/var/cpaas` from the capacity pages before the first apply (see + [Size the clusters](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#size-the-clusters)). +- **`diskGiB` is the system disk only.** Data disks are declared in the pool; adding their sizes to + `diskGiB` doubles the capacity. The templates use `cloneMode: fullClone` so that `diskGiB` takes effect — + a linked clone keeps the template's disk size and ignores it. `diskGiB` must be at least the template's + system disk. +- **Pod logs move onto the ephemeral containerd disk** and are lost with the VM. Logs you need to keep are + the log stack's job, not the node's. + +#### Rollouts and immutable templates + +- **No surge machine exists in a fixed-slot pool.** The `KubeadmControlPlane` uses + `spec.rolloutStrategy.rollingUpdate.maxSurge: 0` with three replicas; a `MachineDeployment` uses + `spec.strategy.rollingUpdate.maxSurge: 0` with `maxUnavailable: 1`. The provider's admission webhooks + reject anything else. The two field paths differ — `KubeadmControlPlane` has no `strategy` and no + `maxUnavailable` — and the Cluster API default `maxUnavailable: 0` deadlocks against `maxSurge: 0`. +- **Templates are immutable.** A change to VM size, VM template, network devices or bootstrap + configuration is a new `VSphereMachineTemplate` or `KubeadmConfigTemplate` under a new name + (`-infra-v2`) plus a reference switch, which rolls the pool node by node. Rolling back is + switching the reference back. See + [Roll out updated worker node configuration](https://docs.alauda.io/immutable-infra/1.0/manage-nodes/vmware-vsphere.html#roll-out-updated-worker-node-configuration). +- **The provider owns some `KubeadmControlPlane` fields.** It adds the CoreDNS and kube-proxy skip + annotations and sets `dns.imageRepository`. Set `dns.imageTag` with its `-v4.4.x` suffix and never set + `dns.imageRepository` — the provider reads the suffix to choose the repository. Never re-apply or + replace the stored control-plane manifest after creation; change the control plane in place with + `kubectl edit` or `kubectl patch`, as the product upgrade and node procedures do. See + [Provider-managed fields on your manifests](https://docs.alauda.io/immutable-infra/1.0/create-cluster/vmware-vsphere.html#provider-managed-fields). +- **Operate VMs through Cluster API, not vCenter.** Since `v1.0.17` a VM powered off in vCenter stays off + and its Machine reports not ready; power it on in vCenter to recover. + +#### vSphere CPI through `ClusterResourceSet` + +The vSphere cloud provider (CPI) is delivered by a `ClusterResourceSet` on `global` as soon as the +workload API server answers; it sets each node's `providerID`. Two delivery rules: + +- The `ClusterResourceSet`, its ConfigMaps and its Secret live in `cpaas-system`, the namespace of the + `Cluster` — a `ClusterResourceSet` in another namespace silently delivers nothing. +- **The selector is scoped to one cluster.** The product example selects on + `addons.cluster.x-k8s.io/vsphere-cpi: "enabled"`, which every vSphere cluster carries; with several + workload clusters, every cluster's `ClusterResourceSet` would bind to every cluster and push its own + credentials and CPI configuration into all of them. This delivery adds + `cluster.x-k8s.io/cluster-name: ` — a label the product manifest already sets on the + `Cluster` — to the selector; `matchLabels` entries must all match, so the set binds to its own + cluster only. A set created earlier with the product selector still binds to every new cluster, which + is why [Check before applying](#check-before-applying) looks for one. + +#### Failure domains + +The main line is one datacenter, one port group and one NIC per node. To spread nodes across +datacenters or compute clusters, use `VSphereFailureDomain` and `VSphereDeploymentZone` — they belong +together with `VSphereCluster.spec.failureDomainSelector` and the CPI `[Labels]` block as one +configuration set; enabling only part of it leaves nodes with unresolvable zone labels or a control +plane without a placement target. The template, port group and datastore names must resolve in every +datacenter. Follow +[Multiple datacenters and failure domains](https://docs.alauda.io/immutable-infra/1.0/create-cluster/vmware-vsphere.html#multiple-datacenters-and-failure-domains) +and introduce one topology variant at a time. + +#### Secrets and keys + +- **vCenter credentials** go into two Secrets per cluster — the identity referenced by `VSphereCluster` + and the CPI credential — both created with `kubectl create` from values typed at a prompt. Never + `kubectl apply` a Secret that carries a credential: client-side apply stores the full manifest in the + `last-applied-configuration` annotation of the Secret itself. +- **Encryption at rest.** The product example writes the AES key inline into the `KubeadmControlPlane`. + This delivery keeps the same `kubeadmConfigSpec.files` entry but fills it with + `contentFrom.secret` — the Kubeadm Provider mechanism the product also uses for the Bare Metal + ServiceAccount key files + ([Prepare Shared ServiceAccount Signing Key](https://docs.alauda.io/immutable-infra/1.0/global/install.html#prepare-shared-serviceaccount-signing-key)) — + so the key appears in no parameter file, manifest or archive. Generate the key once: a later + control-plane node with a different key cannot decrypt the Secrets already stored in etcd. Keep the + Secret on whichever `global` is active; a new control-plane machine cannot bootstrap without it. + +#### Kubernetes 1.35 kubelet setting + +ACP 4.4 ships Kubernetes 1.35, so both kubelet patches carry +`"imagePullCredentialsVerificationPolicy": "NeverVerify"`, as the product page requires for 1.35 — its +baseline manifests omit the field because they are written for Kubernetes 1.34 and earlier. Without it, +the 1.35 kubelet can verify pull credentials even for the images preloaded in the template. See +[Required kubelet patch for Kubernetes 1.35](https://docs.alauda.io/immutable-infra/1.0/upgrade-cluster/index.html#kubernetes-1-35-kubelet-patch). + +### Parameters + +Each workload cluster has its own cluster file, `wl-.env`, next to the shared `site.env` — see +[Prepare the parameter files](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#prepare-the-parameter-files). +The common variables (`CLUSTER_NAME`, `CONTROL_PLANE_VIP`, `CLUSTER_CIDR`, `SERVICE_CIDR`, +`KUBE_OVN_JOIN_CIDR`, `INGRESS_VIP_RANGE`, `GLOBAL_KUBECONFIG`, `CLUSTER_KUBECONFIG`, `K8S_VERSION`, +`BOOT_SSH_PUBLIC_KEY`, `PLATFORM_HOST`, `REGISTRY_DOMAIN`, `ACP_TOKEN`, …) and the helpers `render`, +`load_env` and `apply_rendered` are defined there. A vSphere workload cluster file adds the following. + +| Variable | Meaning | Example | Where to get it | +| --- | --- | --- | --- | +| `CONTROL_PLANE_VRID` | Keepalived VRID of the Self-built VIP, `1`–`255` | `51` | Network team; unique on the control-plane Layer-2 segment | +| `VSPHERE_SERVER` | vCenter address (FQDN or IP) as reached from `global` and the workload nodes | `vcenter.example.com` | vSphere administrator | +| `VSPHERE_THUMBPRINT` | SHA-1 fingerprint of the vCenter certificate | `00:11:22:33:44:55:66:77:88:99:AA:BB:CC:DD:EE:FF:00:11:22:33` | `openssl` command in [Collect the vCenter facts](#collect-the-vcenter-facts) | +| `VSPHERE_DATACENTER` | vCenter datacenter of the nodes | `dc-a` | `govc find / -type d`, confirmed by the vSphere administrator | +| `VSPHERE_DATASTORE` | Datastore for the VM system disks | `datastore-a` | `govc find / -type s`, confirmed by the vSphere administrator | +| `VSPHERE_FOLDER` | VM folder of this cluster, `//vm/` | `/dc-a/vm/wl-prod` | Created in vCenter before the first apply | +| `VSPHERE_NETWORK` | Port group of the node NIC | `pg-wl-prod` | `govc find / -type n`, confirmed by the network team | +| `VSPHERE_TEMPLATE` | VM template created from the Alauda OS OVA | `alauda-os-4.4.0` | Name given when the OVA was imported | +| `VSPHERE_CPI_IMAGE_TAG` | Tag of `ait/cloud-provider-vsphere` | `v1.33.1-alauda.1` | Registry tag list after the provider package is uploaded | +| `CLUSTER_REGISTRY_ADDRESS` | Permanent platform registry, `:`, for `cpaas.io/registry-address` | `acp.example.com:11443` | `cpaas.io/registry-address` annotation of the `global` `Cluster`; normally the value of `REGISTRY_DOMAIN` | +| `IMAGE_REGISTRY_ADDRESS` | Registry of the kubeadm `imageRepository` and the CPI image | `acp.example.com:11443` | Normally the same as `CLUSTER_REGISTRY_ADDRESS` | +| `KUBE_OVN_VERSION` | Kube-OVN chart version | `v4.4.6` | OS Support Matrix row of the delivered ACP release, column *kube-ovn (chart)* | +| `DNS_IMAGE_TAG` | CoreDNS tag, copied with its `-v4.4.x` suffix | `1.14.3-v4.4.6` | Same row, column *coredns* | +| `ETCD_IMAGE_TAG` | etcd tag | `v3.6.13-acp.0` | Same row, column *etcd* | +| `NODE_PREFIX` | Prefix length of the node subnet | `24` | Network team | +| `NODE_GATEWAY` | Default gateway of the node subnet | `192.0.2.1` | Network team | +| `NODE_DNS` | DNS server written to the nodes' `/etc/resolv.conf` | `203.0.113.53` | Customer DNS; must resolve the platform domain and, when `VSPHERE_SERVER` is a name, the vCenter | +| `CP_NODE_IP_1` | Control-plane node 1 IP, without prefix | `192.0.2.11` | IP plan | +| `CP_NODE_IP_2` | Control-plane node 2 IP | `192.0.2.12` | IP plan | +| `CP_NODE_IP_3` | Control-plane node 3 IP | `192.0.2.13` | IP plan | +| `INFRA_NODE_IP_1` | `infra` node 1 IP | `192.0.2.21` | IP plan | +| `INFRA_NODE_IP_2` | `infra` node 2 IP | `192.0.2.22` | IP plan | +| `INFRA_NODE_IP_3` | `infra` node 3 IP | `192.0.2.23` | IP plan | +| `CP_NUM_CPUS` | vCPUs per control-plane node | `4` | [Size the clusters](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#size-the-clusters) | +| `CP_MEMORY_MIB` | Memory per control-plane node, MiB | `16384` | Same | +| `CP_SYSTEM_DISK_GIB` | Control-plane system disk, GiB | `300` | At least the template's system disk; `300` is the product's baseline example | +| `CP_VAR_CPAAS_SIZE_GIB` | Control-plane `/var/cpaas` (persistent), GiB | `100` | Product minimum 100 | +| `CP_VAR_LIB_KUBELET_SIZE_GIB` | Control-plane `/var/lib/kubelet` (ephemeral), GiB | `100` | Product minimum 100 | +| `CP_VAR_LIB_CONTAINERD_SIZE_GIB` | Control-plane `/var/lib/containerd` (ephemeral), GiB | `100` | Product minimum 100 | +| `CP_VAR_LIB_ETCD_SIZE_GIB` | Control-plane `/var/lib/etcd` (ephemeral), GiB | `20` | Product minimum 20 | +| `INFRA_NUM_CPUS` | vCPUs per `infra` node | `8` | [Size the clusters](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#size-the-clusters) | +| `INFRA_MEMORY_MIB` | Memory per `infra` node, MiB | `32768` | Same | +| `INFRA_SYSTEM_DISK_GIB` | `infra` system disk, GiB | `300` | At least the template's system disk; `300` is the product's baseline example | +| `INFRA_VAR_CPAAS_SIZE_GIB` | `infra` `/var/cpaas` (persistent), GiB | `100` | Product minimum 100, plus the log and monitoring data from [Size the clusters](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#size-the-clusters) | +| `INFRA_VAR_LIB_KUBELET_SIZE_GIB` | `infra` `/var/lib/kubelet` (ephemeral), GiB | `100` | Product minimum 100 | +| `INFRA_VAR_LIB_CONTAINERD_SIZE_GIB` | `infra` `/var/lib/containerd` (ephemeral), GiB | `100` | Product minimum 100 | +| `LOG_NODE_IP_1` | Dedicated-pool variant only: `log` node 1 IP (one variable per slot, like `INFRA_NODE_IP_`) | `192.0.2.41` | IP plan | +| `LOG_NUM_CPUS` | Dedicated-pool variant only: vCPUs per `log` node | `8` | Size the clusters | +| `LOG_MEMORY_MIB` | Dedicated-pool variant only: memory per `log` node, MiB | `32768` | Size the clusters | +| `LOG_VAR_CPAAS_SIZE_GIB` | Dedicated-pool variant only: `log` `/var/cpaas`, GiB | `100` | Size the clusters | + +Values that never go into a file — typed at a prompt when a command needs them (the platform token `ACP_TOKEN` is defined in the common article): + +| Variable | Meaning | Example | Where to get it | +| --- | --- | --- | --- | +| `VSPHERE_USERNAME` | vCenter account used by the provider and the CPI | `svc-capv@vsphere.example.com` | vSphere administrator | +| `VSPHERE_PASSWORD` | Its password | (typed) | vSphere administrator | + +Append the vSphere variables to the workload cluster file, after the common values: + +```bash +# wl-prod.env (continued) — VMware vSphere variables of workload cluster wl-prod +CONTROL_PLANE_VRID='51' +VSPHERE_SERVER='vcenter.example.com' +VSPHERE_THUMBPRINT='00:11:22:33:44:55:66:77:88:99:AA:BB:CC:DD:EE:FF:00:11:22:33' +VSPHERE_DATACENTER='dc-a' +VSPHERE_DATASTORE='datastore-a' +VSPHERE_FOLDER='/dc-a/vm/wl-prod' +VSPHERE_NETWORK='pg-wl-prod' +VSPHERE_TEMPLATE='alauda-os-4.4.0' +VSPHERE_CPI_IMAGE_TAG='v1.33.1-alauda.1' +CLUSTER_REGISTRY_ADDRESS='acp.example.com:11443' +IMAGE_REGISTRY_ADDRESS='acp.example.com:11443' +KUBE_OVN_VERSION='v4.4.6' +DNS_IMAGE_TAG='1.14.3-v4.4.6' +ETCD_IMAGE_TAG='v3.6.13-acp.0' +NODE_PREFIX='24' +NODE_GATEWAY='192.0.2.1' +NODE_DNS='203.0.113.53' +CP_NODE_IP_1='192.0.2.11' +CP_NODE_IP_2='192.0.2.12' +CP_NODE_IP_3='192.0.2.13' +INFRA_NODE_IP_1='192.0.2.21' +INFRA_NODE_IP_2='192.0.2.22' +INFRA_NODE_IP_3='192.0.2.23' +CP_NUM_CPUS='4' +CP_MEMORY_MIB='16384' +CP_SYSTEM_DISK_GIB='300' +CP_VAR_CPAAS_SIZE_GIB='100' +CP_VAR_LIB_KUBELET_SIZE_GIB='100' +CP_VAR_LIB_CONTAINERD_SIZE_GIB='100' +CP_VAR_LIB_ETCD_SIZE_GIB='20' +INFRA_NUM_CPUS='8' +INFRA_MEMORY_MIB='32768' +INFRA_SYSTEM_DISK_GIB='300' +INFRA_VAR_CPAAS_SIZE_GIB='100' +INFRA_VAR_LIB_KUBELET_SIZE_GIB='100' +INFRA_VAR_LIB_CONTAINERD_SIZE_GIB='100' +``` + +Commands in this article that are not `apply_rendered` calls use the parameters directly; run them in +the delivery directory after `load_env wl-prod.env`. + +### Prepare VMware vSphere + +#### Collect the vCenter facts + +Query vCenter read-only and have the vSphere administrator confirm each selection — a discovered object +is a candidate, not a decision. + +```bash +# on a workstation where govc is already authenticated +govc find / -type d # datacenters -> VSPHERE_DATACENTER +govc find / -type n # port groups -> VSPHERE_NETWORK +govc find / -type s # datastores -> VSPHERE_DATASTORE +govc find / -type m # VMs and templates -> VSPHERE_TEMPLATE (confirm it is a template) +govc find / -type c # compute clusters (failure-domain variant only) +govc find / -type p # resource pools (failure-domain variant only) + +# vCenter certificate fingerprint -> VSPHERE_THUMBPRINT +openssl s_client -connect "${VSPHERE_SERVER}:443" -servername "${VSPHERE_SERVER}" /dev/null \ + | openssl x509 -noout -fingerprint -sha1 +``` + +Also settle with the vSphere administrator: + +- **The vCenter account** (`VSPHERE_USERNAME`) can clone the template and create and manage VMs in the + folder, datastore and port group above, and holds **Global → Manage custom attributes** and + **Global → Set custom attribute** — the provider needs both whenever a `vsphere.cluster.x-k8s.io/*` + annotation is present, and a missing privilege fails VM reconciliation instead of being skipped. See + [VMware vSphere Provider Installation](https://docs.alauda.io/immutable-infra/1.0/install/vmware-vsphere.html#prerequisites). +- **The VM folder** `VSPHERE_FOLDER` exists; one folder per cluster shows in vCenter which VMs belong to it. +- **The port group** lets VRRP and gratuitous ARP through for the Self-built VIP (dvPortGroup security + policy, NSX-T distributed firewall), and the node IPs, `CONTROL_PLANE_VIP` and the MetalLB pool + `INGRESS_VIP_RANGE` (with `INGRESS_VIP`) are reserved in its subnet. +- **The datastore** has room for every slot's disks: system disk, `/var/cpaas` and the ephemeral disks. + Persistent disks of a deleted cluster stay on the datastore until provider-managed reclaim finishes. + +#### Prepare the VM template + +**Import the OVA.** Download the Alauda OS **OVA** for the delivered ACP 4.4 release from the OS download +page in Customer Portal +([image format by provider](https://docs.alauda.io/immutable-infra/1.0/overview/os-support-matrix.html#image-format-by-provider)) +and import it as a template named `VSPHERE_TEMPLATE` in `VSPHERE_DATACENTER` — in every datacenter when +failure domains are used, because the name must resolve in each. + +**Check what it contains.** The template must carry `cloud-init`, VMware Tools or `open-vm-tools`, +`containerd`, the kubeadm baseline components and the preloaded image tar files under `/root/images/`: +static IPs and hostnames depend on `cloud-init`, node IP reporting on the guest tools, and an air-gapped +bootstrap on the preloaded images. For the Self-built VIP, the `ip_vs` module family must be loadable. + +**Validate the preloaded images on a disposable VM** cloned from the template. Run inside that VM: + +```bash +# inside a disposable VM cloned from VSPHERE_TEMPLATE +sudo bash -euo pipefail <<'EOF' +shopt -s nullglob +image_files=(/root/images/*.tar) +if (( ${#image_files[@]} == 0 )); then + echo "ERROR: no tar files found in /root/images" >&2 + exit 1 +fi +systemctl restart containerd +for image_file in "${image_files[@]}"; do + ctr -n k8s.io images import "${image_file}" +done +ctr -n k8s.io images list -q | sort -u +EOF + +sudo grep -nE '^[[:space:]]*(sandbox_image|sandbox)[[:space:]]*=' /etc/containerd/config.toml +``` + +The list must contain kube-apiserver, kube-controller-manager, kube-scheduler and etcd under exactly the +references kubeadm will ask for — `IMAGE_REGISTRY_ADDRESS` + `/tkestack`, with the tags derived from +`K8S_VERSION` and `ETCD_IMAGE_TAG` — and the sandbox image exactly as `containerd` is configured. A +missing or different reference makes the node pull that image from `IMAGE_REGISTRY_ADDRESS` instead; in +an air-gapped site, confirm that the registry serves it under that exact reference, or correct the +template, before you continue. Discard the VM afterwards; never convert it back into the template. Repeat +the check whenever the template, `K8S_VERSION`, `IMAGE_REGISTRY_ADDRESS` or `ETCD_IMAGE_TAG` changes. See +[Validate the locally preloaded images](https://docs.alauda.io/immutable-infra/1.0/infrastructure/vmware-vsphere.html#validate-the-locally-preloaded-images). + +#### Approve the network flows + +| From | To | Port | Why | +| --- | --- | --- | --- | +| `global` nodes | vCenter (`VSPHERE_SERVER`) | TCP `443` | The provider clones and manages the VMs | +| Workload nodes | vCenter (`VSPHERE_SERVER`) | TCP `443` | The vSphere CPI runs on the workload nodes (`port = "443"` in its configuration) | +| `global` | Workload control-plane nodes and `CONTROL_PLANE_VIP` | TCP `6443`, `11780` | Management of the workload cluster — see [Connecting `global` Cluster and Workload Cluster](https://docs.alauda.io/container_platform/4.4/configure/clusters/on-premises.html#connecting-global-cluster-and-workload-cluster) | +| Workload nodes | `global` registry (`CLUSTER_REGISTRY_ADDRESS`) and platform address (`PLATFORM_HOST`) | TCP `11443`, `443` | Platform images, plugins and platform access | +| Workload control-plane nodes | Each other | VRRP (IP protocol `112`), gratuitous ARP | Self-built VIP | +| Workload nodes | Customer DNS and NTP | `53`, `123` | Name resolution and time — see [Configure machines](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#configure-machines) | + +These flows add to the ones in +[Reference design](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#reference-design). +On Alauda OS nodes the platform maintains the node port configuration, so the approvals above concern +firewalls between networks. For anything not listed, use the +[Communication Matrix](https://docs.alauda.io/container_platform/4.4/install/prepare/communication_matrix.html#not_covered) +of ACP 4.4 and the pages it points to — do not copy its tables into the change request. + +#### Prepare the `global` cluster + +Workload cluster objects are created on `global` in `cpaas-system`, and the vSphere provider there does +the work. A `global` cluster on DCS, HCS or Bare Metal does not carry the vSphere provider until you +install it. + +**1. Check the providers.** + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get minfo -l cpaas.io/module-name=cluster-api-provider-vsphere +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get minfo -l cpaas.io/module-name=cluster-api-provider-kubeadm +``` + +Both must exist and be `Running`. If the vSphere provider is missing, download +**Alauda Container Platform VMware vSphere Infrastructure Provider** (`v1.0.17`) — and +**Alauda Container Platform Kubeadm Provider** only if it is missing too — from Customer Portal into +`packages/vsphere-provider/`, push that directory with `violet` (the account needs the +`platform-admin-system` role), and install the plugins on `global` from **Administrator** > +**Marketplace** > **Cluster Plugins** (or with a `ModuleInfo`, see +[Installing via YAML](https://docs.alauda.io/container_platform/4.4/extend/cluster_plugin.html#installing-via-yaml)). +The vSphere provider package also carries the CPI image. + +```bash +# on the delivery host — primary global, while the platform domain points at it +load_env wl-prod.env +read -rsp 'Platform API token: ' ACP_TOKEN; echo +violet push packages/vsphere-provider --platform-address "https://${PLATFORM_HOST}" --platform-token "${ACP_TOKEN}" +unset ACP_TOKEN +``` + +**2. Verify the provider installation** — nine `vsphere*` CRDs and the admission webhooks. + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get crd | grep -E '^vsphere[a-z]+\.infrastructure\.cluster\.x-k8s\.io\b' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get validatingwebhookconfiguration \ + -o jsonpath='{range .items[*]}{range .webhooks[*]}{.name}{"\n"}{end}{end}' \ + | grep -E 'vsphere|capv' +``` + +Expect `validation.kubeadmcontrolplane.capv.cluster.x-k8s.io` and +`validation.machinedeployment.capv.cluster.x-k8s.io` among the five webhooks: they reject a wrong rollout +setting up front. Without them a wrong value is not rejected — the rollout just stalls for want of a free +slot. See +[Verify the installation](https://docs.alauda.io/immutable-infra/1.0/install/vmware-vsphere.html#verify-the-installation). + +**3. Check the management controller, registry and DR role.** + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get deploy capi-controller-manager \ + -o jsonpath='{.spec.template.spec.containers[0].args}{"\n"}' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get secret public-registry-credential -o name +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get cluster global \ + -o jsonpath='{.metadata.annotations.cpaas\.io/registry-address}{"\n"}' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n kube-public get configmap etcd-sync -o name 2>/dev/null || echo "not standby" + +curl -sk "https://${CLUSTER_REGISTRY_ADDRESS}/v2/acp/chart-kube-ovn/tags/list" +curl -sk "https://${IMAGE_REGISTRY_ADDRESS}/v2/acp/coredns/tags/list" +curl -sk "https://${IMAGE_REGISTRY_ADDRESS}/v2/ait/cloud-provider-vsphere/tags/list" +``` + +Pass criteria: + +- The controller arguments contain `ClusterResourceSet=true` — without it the CPI is never delivered. +- They also contain `--additional-sync-machine-labels` covering `node-role.kubernetes.io/`. Pool labels + reach the Node only through it; without it every plugin `nodeSelector` stays `Pending`, which looks + exactly like a wrong label. If it is missing, enable it now as described under *Enable label + synchronization* in + [Install the platform plugins](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#install-the-platform-plugins) + (product reference: + [Configure Annotations, Labels, and Taints on Nodes](https://docs.alauda.io/immutable-infra/1.0/how-to/node-configuration.html)) — + the Node label check in step 6 of [Apply and verify](#apply-and-verify) depends on it. +- `public-registry-credential` exists (its content is not printed) and the registry annotation equals + `CLUSTER_REGISTRY_ADDRESS` — the workload cluster inherits this registry. A dedicated registry per + workload cluster is decided now or never; see + [Choose the Image Registry for a Workload Cluster](https://docs.alauda.io/immutable-infra/1.0/how-to/workload-cluster-image-registry.html). +- `not standby` is printed. On a disaster-recovery standby the provider skips every object that does not + belong to `global`, so a creation there looks stalled rather than failed. Create workload clusters on + the active `global` only. +- The tag lists contain `KUBE_OVN_VERSION`, `DNS_IMAGE_TAG` and `VSPHERE_CPI_IMAGE_TAG`. A `NAME_UNKNOWN` + answer means a wrong repository name, not a missing tag. An authenticated registry answers + `UNAUTHORIZED`; repeat with `-u ` and let `curl` prompt for the password. + +**4. Check the `alive` plugin** that the Self-built VIP depends on. + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get clustermodule global -o jsonpath='{.spec.version}{"\n"}' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get moduleplugin alive -o jsonpath='{.status.targetClusterVersions}{"\n"}' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get moduleplugin alive -o jsonpath='{.status.latestVersion}{"\n"}' +``` + +Take the `alive` version from the `targetClusterVersions` entry of the printed `ClusterModule` version +(`latestVersion` only when there is no entry), then +`kubectl get moduleconfig alive- -o jsonpath='{.status.readyForDeploy}{"\n"}'` must print +`true`. If the plugin is absent or not ready, the Self-built VIP cannot be installed: obtain the matching +plugin package from Alauda support, or use the external load balancer variant. See +[Self-built VIP prerequisites](https://docs.alauda.io/immutable-infra/1.0/create-cluster/vmware-vsphere.html#self-built-vip-prerequisites). + +**5. With a `global` DR pair**, install the same provider version on the standby too, so that the standby +can manage the vSphere workload clusters after a failover. Each side has its own registry: push +`packages/vsphere-provider` to the standby through its own VIP and registry, exactly as +[Install the platform plugins](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#install-the-platform-plugins) +uploads `packages/plugins` there (see also +[Upload an Operator to a Standby Global Cluster](https://docs.alauda.io/container_platform/4.4/extend/upload_package.html#upload-an-operator-to-a-standby-global-cluster)). +While `etcd-sync` runs on the standby, its provider leaves the workload clusters alone (see +[Standby Management Clusters](https://docs.alauda.io/immutable-infra/1.0/install/vmware-vsphere.html#standby-management-cluster)). +After a failover, confirm on the new active side that the three Secrets of every vSphere workload +cluster ([Create the credential Secrets](#create-the-credential-secrets)) exist before any node of that +cluster is replaced: the provider cannot reach vCenter without the first, and a new control-plane node +cannot bootstrap without the third. + +#### Create the credential Secrets + +Three Secrets per workload cluster, all in `cpaas-system` on `global`, all created with `kubectl create`. +The values are typed at a prompt and passed through bash process substitution, so they reach neither a +file, the shell history nor the process list. + +```bash +# on global, in bash with the parameter files loaded +: "${CLUSTER_NAME:?load the parameter files first}" "${VSPHERE_SERVER:?load the parameter files first}" +read -rp 'vCenter username: ' VSPHERE_USERNAME +read -rsp 'vCenter password: ' VSPHERE_PASSWORD; echo + +# 1. Identity referenced by VSphereCluster.spec.identityRef +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system create secret generic "${CLUSTER_NAME}-vsphere-credentials" \ + --from-file=username=<(printf '%s' "${VSPHERE_USERNAME}") \ + --from-file=password=<(printf '%s' "${VSPHERE_PASSWORD}") + +# 2. CPI credential, delivered into the workload cluster by the ClusterResourceSet +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system create secret generic "${CLUSTER_NAME}-vsphere-cpi-secret" \ + --type=addons.cluster.x-k8s.io/resource-set \ + --from-file=data=<(printf '%s\n' \ + 'apiVersion: v1' \ + 'kind: Secret' \ + 'metadata:' \ + ' name: vsphere-cloud-secret' \ + ' namespace: kube-system' \ + 'type: Opaque' \ + 'data:' \ + " ${VSPHERE_SERVER}.username: $(printf '%s' "${VSPHERE_USERNAME}" | base64 | tr -d '\n')" \ + " ${VSPHERE_SERVER}.password: $(printf '%s' "${VSPHERE_PASSWORD}" | base64 | tr -d '\n')") + +unset VSPHERE_USERNAME VSPHERE_PASSWORD + +# 3. Encryption-at-rest configuration, generated once; the key exists only in this Secret +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system create secret generic "${CLUSTER_NAME}-encryption-provider-config" \ + --from-file=encryption-provider.conf=<(printf '%s\n' \ + 'apiVersion: apiserver.config.k8s.io/v1' \ + 'kind: EncryptionConfiguration' \ + 'resources:' \ + '- resources:' \ + ' - secrets' \ + ' providers:' \ + ' - aescbc:' \ + ' keys:' \ + ' - name: key1' \ + " secret: $(head -c 32 /dev/urandom | base64 | tr -d '\n')") +``` + +- The CPI payload uses base64 `data` rather than the product example's `stringData`, so a password + containing YAML-significant characters (`:`, `#`, quotes, a leading `*`) cannot break it. The Secret's + only key must be `data`. +- If a command reports `AlreadyExists`, keep the existing Secret. Never delete or recreate the + encryption Secret while the cluster exists: every new control-plane node reads it, and a new key + cannot decrypt the data already in etcd. + +### Create a workload cluster + +All objects are created on `global`; the workload cluster's own kubeconfig is used only for checks. Each +file is applied on its own with `apply_rendered` from +[Prepare the parameter files](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#prepare-the-parameter-files), +which refuses an unreplaced `${...}` and lets the server-side admission webhooks judge the file before it +is applied — a client-side dry run contacts no server and would not catch, for example, an out-of-range +`vrid`. **Never apply the directory at once**: each step's check is what catches a wrong value while it +is still cheap to fix. Templates and their controllers are separate files so that the controller's +server-side check finds the template it references. There is no namespace manifest: the objects go to +`cpaas-system`, which always exists on `global`. + +```text +wl-prod/ +├── 10-cluster.yaml # Cluster + VSphereCluster (Self-built VIP) +├── 15-vsphere-cpi.yaml # CPI ConfigMaps + ClusterResourceSet +├── 20-control-plane-pool.yaml # VSphereMachineConfigPool, 3 control-plane slots +├── 21-control-plane-template.yaml # VSphereMachineTemplate +├── 22-control-plane.yaml # KubeadmControlPlane +├── 30-infra-pool.yaml # VSphereMachineConfigPool, infra slots +├── 31-infra-templates.yaml # VSphereMachineTemplate + KubeadmConfigTemplate +└── 32-infra.yaml # MachineDeployment +``` + +#### Check before applying + +These are the mistakes that otherwise surface as half-provisioned machines or a cluster that has to be +rebuilt: + +- `CLUSTER_NAME` is not `global` and does not start with `global-` (both are reserved for the `global` + cluster's DR workflow, which selects resources by that prefix), is a valid DNS-1123 label, and is short + enough that `-infra-3` still is one. +- Pod, Service and Join CIDRs overlap neither each other nor the node subnet, the VIPs, the vCenter + networks, `global`'s CIDRs, other workload clusters' CIDRs or customer networks — nothing checks this + at creation. +- `CONTROL_PLANE_VIP` is an unused IPv4 address in the node subnet and not one of the slot IPs; + `CONTROL_PLANE_VRID` is unused on the segment. +- Every slot IP is unused, and every hostname is unused in the vCenter. +- `CP_SYSTEM_DISK_GIB` and `INFRA_SYSTEM_DISK_GIB` are at least the template's system disk. +- The three Secrets exist — the command prints names only, never contents. +- No existing `ClusterResourceSet` selects on `addons.cluster.x-k8s.io/vsphere-cpi` without also + selecting its own `cluster.x-k8s.io/cluster-name`. Such a set — one created with the product's + example selector — also binds to the new cluster and pushes another cluster's CPI configuration and + vCenter credential into it. Resolve it with the platform team before step 1. + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get secret "${CLUSTER_NAME}-vsphere-credentials" \ + "${CLUSTER_NAME}-vsphere-cpi-secret" "${CLUSTER_NAME}-encryption-provider-config" -o name +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get clusterresourceset \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.clusterSelector.matchLabels}{"\n"}{end}' +``` + +#### Manifests + +`wl-prod/10-cluster.yaml` — the cluster root and the vCenter connection. The `Cluster` annotations are +read by platform controllers, which install Kube-OVN and bind the registry; the registry binding is read +once, at creation. + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: ${CLUSTER_NAME} + namespace: cpaas-system + labels: + cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME} + cluster-type: VSphere + addons.cluster.x-k8s.io/vsphere-cpi: "enabled" + annotations: + capi.cpaas.io/resource-group-version: infrastructure.cluster.x-k8s.io/v1beta1 + capi.cpaas.io/resource-kind: VSphereCluster + cpaas.io/sentry-deploy-type: Baremetal + cpaas.io/alb-address-type: ClusterAddress + cpaas.io/network-type: kube-ovn + cpaas.io/kube-ovn-version: ${KUBE_OVN_VERSION} + cpaas.io/kube-ovn-join-cidr: ${KUBE_OVN_JOIN_CIDR} + cpaas.io/registry-address: ${CLUSTER_REGISTRY_ADDRESS} +spec: + clusterNetwork: + pods: + cidrBlocks: + - ${CLUSTER_CIDR} + services: + cidrBlocks: + - ${SERVICE_CIDR} + controlPlaneRef: + apiVersion: controlplane.cluster.x-k8s.io/v1beta1 + kind: KubeadmControlPlane + name: ${CLUSTER_NAME}-kcp + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: VSphereCluster + name: ${CLUSTER_NAME} +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: VSphereCluster +metadata: + name: ${CLUSTER_NAME} + namespace: cpaas-system +spec: + controlPlaneEndpoint: + host: "${CONTROL_PLANE_VIP}" + port: 6443 + controlPlaneLoadBalancer: # write-once: decided at creation, never changed + type: internal + host: "${CONTROL_PLANE_VIP}" # must equal controlPlaneEndpoint.host + port: 6443 # must equal controlPlaneEndpoint.port + vrid: ${CONTROL_PLANE_VRID} + identityRef: + kind: Secret + name: ${CLUSTER_NAME}-vsphere-credentials + server: "${VSPHERE_SERVER}" + thumbprint: "${VSPHERE_THUMBPRINT}" +``` + +`wl-prod/15-vsphere-cpi.yaml` — the CPI configuration and manifests, delivered by a `ClusterResourceSet` +that matches this cluster only. `insecure-flag = "1"` lets the CPI connect when the nodes do not trust the +vCenter certificate. Add a `[Labels]` block only together with failure domains. + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: ${CLUSTER_NAME}-vsphere-cpi-config + namespace: cpaas-system +data: + data: | + apiVersion: v1 + kind: ConfigMap + metadata: + name: cloud-config + namespace: kube-system + data: + vsphere.conf: | + [Global] + secret-name = "vsphere-cloud-secret" + secret-namespace = "kube-system" + service-account = "cloud-controller-manager" + port = "443" + insecure-flag = "1" + datacenters = "${VSPHERE_DATACENTER}" + + [VirtualCenter "${VSPHERE_SERVER}"] +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: ${CLUSTER_NAME}-vsphere-cpi-manifests + namespace: cpaas-system +data: + data: | + apiVersion: v1 + kind: ServiceAccount + metadata: + name: cloud-controller-manager + namespace: kube-system + automountServiceAccountToken: false + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: system:cloud-controller-manager + rules: + - apiGroups: [""] + resources: ["events"] + verbs: ["create", "patch", "update"] + - apiGroups: [""] + resources: ["nodes"] + verbs: ["*"] + - apiGroups: [""] + resources: ["nodes/status"] + verbs: ["patch"] + - apiGroups: [""] + resources: ["services"] + verbs: ["list", "patch", "update", "watch"] + - apiGroups: [""] + resources: ["services/status"] + verbs: ["patch"] + - apiGroups: [""] + resources: ["serviceaccounts"] + verbs: ["create", "get", "list", "watch", "update"] + - apiGroups: [""] + resources: ["persistentvolumes"] + verbs: ["get", "list", "update", "watch"] + - apiGroups: [""] + resources: ["endpoints"] + verbs: ["create", "get", "list", "watch", "update"] + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "list", "watch"] + - apiGroups: ["coordination.k8s.io"] + resources: ["leases"] + verbs: ["get", "list", "watch", "create", "update"] + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: RoleBinding + metadata: + name: servicecatalog.k8s.io:apiserver-authentication-reader + namespace: kube-system + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: extension-apiserver-authentication-reader + subjects: + - apiGroup: "" + kind: ServiceAccount + name: cloud-controller-manager + namespace: kube-system + - apiGroup: "" + kind: User + name: cloud-controller-manager + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: system:cloud-controller-manager + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: system:cloud-controller-manager + subjects: + - kind: ServiceAccount + name: cloud-controller-manager + namespace: kube-system + - kind: User + name: cloud-controller-manager + --- + apiVersion: apps/v1 + kind: DaemonSet + metadata: + annotations: + scheduler.alpha.kubernetes.io/critical-pod: "" + labels: + component: cloud-controller-manager + tier: control-plane + k8s-app: vsphere-cloud-controller-manager + name: vsphere-cloud-controller-manager + namespace: kube-system + spec: + selector: + matchLabels: + k8s-app: vsphere-cloud-controller-manager + updateStrategy: + type: RollingUpdate + template: + metadata: + labels: + component: cloud-controller-manager + k8s-app: vsphere-cloud-controller-manager + spec: + securityContext: + runAsUser: 1001 + automountServiceAccountToken: true + # The platform syncs "global-registry-auth" into the workload cluster when the + # global registry is authenticated (public-registry-credential). Remove these + # two lines when the registry allows anonymous pulls. + imagePullSecrets: + - name: global-registry-auth + serviceAccountName: cloud-controller-manager + hostNetwork: true + tolerations: + - operator: Exists + - key: node.cloudprovider.kubernetes.io/uninitialized + value: "true" + effect: NoSchedule + - key: node-role.kubernetes.io/master + effect: NoSchedule + - key: node.kubernetes.io/not-ready + effect: NoSchedule + operator: Exists + containers: + - name: vsphere-cloud-controller-manager + image: ${IMAGE_REGISTRY_ADDRESS}/ait/cloud-provider-vsphere:${VSPHERE_CPI_IMAGE_TAG} + args: + - --v=2 + - --cloud-provider=vsphere + - --cloud-config=/etc/cloud/vsphere.conf + volumeMounts: + - mountPath: /etc/cloud + name: vsphere-config-volume + readOnly: true + resources: + requests: + cpu: 200m + volumes: + - name: vsphere-config-volume + configMap: + name: cloud-config + --- + apiVersion: v1 + kind: Service + metadata: + labels: + component: cloud-controller-manager + name: vsphere-cloud-controller-manager + namespace: kube-system + spec: + type: NodePort + ports: + - port: 43001 + protocol: TCP + targetPort: 43001 + selector: + component: cloud-controller-manager +--- +apiVersion: addons.cluster.x-k8s.io/v1beta1 +kind: ClusterResourceSet +metadata: + name: ${CLUSTER_NAME}-vsphere-cpi + namespace: cpaas-system +spec: + strategy: Reconcile + clusterSelector: + matchLabels: + addons.cluster.x-k8s.io/vsphere-cpi: "enabled" + cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME} # this cluster only + resources: + - name: ${CLUSTER_NAME}-vsphere-cpi-config + kind: ConfigMap + - name: ${CLUSTER_NAME}-vsphere-cpi-secret + kind: Secret + - name: ${CLUSTER_NAME}-vsphere-cpi-manifests + kind: ConfigMap +``` + +`wl-prod/20-control-plane-pool.yaml` — three control-plane slots. `releaseDelayHours` is the reclaim grace +period of an unused released slot, not a delay before reuse; a replacement in the same pool reuses a +released slot immediately. + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: VSphereMachineConfigPool +metadata: + name: ${CLUSTER_NAME}-cp-pool + namespace: cpaas-system +spec: + clusterRef: + apiVersion: cluster.x-k8s.io/v1beta1 + kind: Cluster + name: ${CLUSTER_NAME} + datacenter: "${VSPHERE_DATACENTER}" + releaseDelayHours: 24 + configs: + - hostname: "${CLUSTER_NAME}-cp-1" + datacenter: "${VSPHERE_DATACENTER}" + network: + primary: + networkName: "${VSPHERE_NETWORK}" + ip: "${CP_NODE_IP_1}/${NODE_PREFIX}" + gateway: "${NODE_GATEWAY}" + dns: + - "${NODE_DNS}" + persistentDisks: + - name: var-cpaas + sizeGiB: ${CP_VAR_CPAAS_SIZE_GIB} + mountPath: /var/cpaas + fsFormat: ext4 + ephemeralDisks: + - name: var-lib-kubelet + sizeGiB: ${CP_VAR_LIB_KUBELET_SIZE_GIB} + mountPath: /var/lib/kubelet + fsFormat: ext4 + - name: var-lib-containerd + sizeGiB: ${CP_VAR_LIB_CONTAINERD_SIZE_GIB} + mountPath: /var/lib/containerd + fsFormat: ext4 + - name: var-lib-etcd + sizeGiB: ${CP_VAR_LIB_ETCD_SIZE_GIB} + mountPath: /var/lib/etcd + fsFormat: ext4 + - hostname: "${CLUSTER_NAME}-cp-2" + datacenter: "${VSPHERE_DATACENTER}" + network: + primary: + networkName: "${VSPHERE_NETWORK}" + ip: "${CP_NODE_IP_2}/${NODE_PREFIX}" + gateway: "${NODE_GATEWAY}" + dns: + - "${NODE_DNS}" + persistentDisks: + - name: var-cpaas + sizeGiB: ${CP_VAR_CPAAS_SIZE_GIB} + mountPath: /var/cpaas + fsFormat: ext4 + ephemeralDisks: + - name: var-lib-kubelet + sizeGiB: ${CP_VAR_LIB_KUBELET_SIZE_GIB} + mountPath: /var/lib/kubelet + fsFormat: ext4 + - name: var-lib-containerd + sizeGiB: ${CP_VAR_LIB_CONTAINERD_SIZE_GIB} + mountPath: /var/lib/containerd + fsFormat: ext4 + - name: var-lib-etcd + sizeGiB: ${CP_VAR_LIB_ETCD_SIZE_GIB} + mountPath: /var/lib/etcd + fsFormat: ext4 + - hostname: "${CLUSTER_NAME}-cp-3" + datacenter: "${VSPHERE_DATACENTER}" + network: + primary: + networkName: "${VSPHERE_NETWORK}" + ip: "${CP_NODE_IP_3}/${NODE_PREFIX}" + gateway: "${NODE_GATEWAY}" + dns: + - "${NODE_DNS}" + persistentDisks: + - name: var-cpaas + sizeGiB: ${CP_VAR_CPAAS_SIZE_GIB} + mountPath: /var/cpaas + fsFormat: ext4 + ephemeralDisks: + - name: var-lib-kubelet + sizeGiB: ${CP_VAR_LIB_KUBELET_SIZE_GIB} + mountPath: /var/lib/kubelet + fsFormat: ext4 + - name: var-lib-containerd + sizeGiB: ${CP_VAR_LIB_CONTAINERD_SIZE_GIB} + mountPath: /var/lib/containerd + fsFormat: ext4 + - name: var-lib-etcd + sizeGiB: ${CP_VAR_LIB_ETCD_SIZE_GIB} + mountPath: /var/lib/etcd + fsFormat: ext4 +``` + +`wl-prod/21-control-plane-template.yaml` — the control-plane VM. A later change to it is a new template +under a new name, never an edit of this one. + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: VSphereMachineTemplate +metadata: + name: ${CLUSTER_NAME}-control-plane + namespace: cpaas-system +spec: + template: + spec: + server: "${VSPHERE_SERVER}" + template: "${VSPHERE_TEMPLATE}" + cloneMode: fullClone + folder: "${VSPHERE_FOLDER}" + datastore: "${VSPHERE_DATASTORE}" + diskGiB: ${CP_SYSTEM_DISK_GIB} + memoryMiB: ${CP_MEMORY_MIB} + numCPUs: ${CP_NUM_CPUS} + os: Linux + powerOffMode: trySoft + network: + devices: + - networkName: "${VSPHERE_NETWORK}" + machineConfigPoolRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: VSphereMachineConfigPool + name: ${CLUSTER_NAME}-cp-pool + namespace: cpaas-system +``` + +`wl-prod/22-control-plane.yaml` — the `KubeadmControlPlane`. `{{ ds.meta_data.hostname }}` and +`{{ local_hostname }}` are cloud-init expressions and the `$` in the image-import script are shell — both +stay literal; `render` substitutes only the parameter-file variables. The encryption configuration comes +from the Secret created earlier. + +```yaml +apiVersion: controlplane.cluster.x-k8s.io/v1beta1 +kind: KubeadmControlPlane +metadata: + name: ${CLUSTER_NAME}-kcp + namespace: cpaas-system +spec: + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + version: "${K8S_VERSION}" + replicas: 3 + machineTemplate: + nodeDrainTimeout: 1m + nodeDeletionTimeout: 5m + metadata: + labels: + node-role.kubernetes.io/control-plane: "" + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: VSphereMachineTemplate + name: ${CLUSTER_NAME}-control-plane + kubeadmConfigSpec: + users: + - name: boot + sudo: ALL=(ALL) NOPASSWD:ALL + sshAuthorizedKeys: + - "${BOOT_SSH_PUBLIC_KEY}" + files: + - path: /etc/resolv.conf + owner: "root:root" + append: false + permissions: "0644" + content: | + nameserver ${NODE_DNS} + - path: /etc/kubernetes/admission/psa-config.yaml + owner: "root:root" + permissions: "0644" + content: | + apiVersion: apiserver.config.k8s.io/v1 + kind: AdmissionConfiguration + plugins: + - name: PodSecurity + configuration: + apiVersion: pod-security.admission.config.k8s.io/v1 + kind: PodSecurityConfiguration + defaults: + enforce: "privileged" + enforce-version: "latest" + audit: "baseline" + audit-version: "latest" + warn: "baseline" + warn-version: "latest" + exemptions: + usernames: [] + runtimeClasses: [] + namespaces: + - kube-system + - cpaas-system + - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json + owner: "root:root" + permissions: "0644" + content: | + { + "apiVersion": "kubelet.config.k8s.io/v1beta1", + "kind": "KubeletConfiguration", + "imagePullCredentialsVerificationPolicy": "NeverVerify", + "protectKernelDefaults": true, + "streamingConnectionIdleTimeout": "5m", + "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt", + "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key" + } + - path: /etc/kubernetes/encryption-provider.conf + owner: "root:root" + append: false + permissions: "0644" + contentFrom: + secret: + name: ${CLUSTER_NAME}-encryption-provider-config + key: encryption-provider.conf + - path: /etc/kubernetes/audit/policy.yaml + owner: "root:root" + append: false + permissions: "0644" + content: | + apiVersion: audit.k8s.io/v1 + kind: Policy + omitStages: + - "RequestReceived" + rules: + - level: None + users: + - system:kube-controller-manager + - system:kube-scheduler + - system:serviceaccount:kube-system:endpoint-controller + verbs: ["get", "update"] + namespaces: ["kube-system"] + resources: + - group: "" + resources: ["endpoints"] + - level: None + nonResourceURLs: + - /healthz* + - /version + - /swagger* + - level: None + resources: + - group: "" + resources: ["events"] + - level: None + resources: + - group: "devops.alauda.io" + - level: None + verbs: ["get", "list", "watch"] + - level: None + resources: + - group: "coordination.k8s.io" + resources: ["leases"] + - level: None + resources: + - group: "authorization.k8s.io" + resources: ["subjectaccessreviews", "selfsubjectaccessreviews"] + - group: "authentication.k8s.io" + resources: ["tokenreviews"] + - level: None + resources: + - group: "app.alauda.io" + resources: ["imagewhitelists"] + - group: "k8s.io" + resources: ["namespaceoverviews"] + - level: Metadata + resources: + - group: "" + resources: ["secrets", "configmaps"] + - level: Metadata + resources: + - group: "operator.connectors.alauda.io" + resources: ["installmanifests"] + - group: "operators.katanomi.dev" + resources: ["katanomis"] + - level: RequestResponse + resources: + - group: "" + - group: "aiops.alauda.io" + - group: "apps" + - group: "app.k8s.io" + - group: "authentication.istio.io" + - group: "auth.alauda.io" + - group: "autoscaling" + - group: "asm.alauda.io" + - group: "clusterregistry.k8s.io" + - group: "crd.alauda.io" + - group: "infrastructure.alauda.io" + - group: "monitoring.coreos.com" + - group: "operators.coreos.com" + - group: "networking.istio.io" + - group: "extensions.istio.io" + - group: "install.istio.io" + - group: "security.istio.io" + - group: "telemetry.istio.io" + - group: "opentelemetry.io" + - group: "networking.k8s.io" + - group: "portal.alauda.io" + - group: "rbac.authorization.k8s.io" + - group: "storage.k8s.io" + - group: "tke.cloud.tencent.com" + - group: "devopsx.alauda.io" + - group: "core.katanomi.dev" + - group: "deliveries.katanomi.dev" + - group: "integrations.katanomi.dev" + - group: "artifacts.katanomi.dev" + - group: "builds.katanomi.dev" + - group: "versioning.katanomi.dev" + - group: "sources.katanomi.dev" + - group: "tekton.dev" + - group: "operator.tekton.dev" + - group: "eventing.knative.dev" + - group: "flows.knative.dev" + - group: "messaging.knative.dev" + - group: "operator.knative.dev" + - group: "sources.knative.dev" + - group: "operator.devops.alauda.io" + - group: "flagger.app" + - group: "jaegertracing.io" + - group: "velero.io" + resources: ["deletebackuprequests"] + - group: "connectors.alauda.io" + - group: "operator.connectors.alauda.io" + resources: ["connectorscores", "connectorsgits", "connectorsocis"] + - level: Metadata + - path: /usr/local/bin/capv-load-local-images.sh + owner: "root:root" + permissions: "0755" + content: | + #!/bin/bash + set -euo pipefail + until mountpoint -q /var/lib/containerd; do + echo "waiting for /var/lib/containerd mount" + sleep 1 + done + systemctl restart containerd + until systemctl is-active --quiet containerd; do + echo "waiting for containerd" + sleep 1 + done + if [ ! -d "/root/images" ]; then + echo "ERROR: /root/images directory not found" >&2 + exit 1 + fi + image_count=0 + for image_file in /root/images/*.tar; do + if [ -f "$image_file" ]; then + echo "importing image: $image_file" + ctr -n k8s.io images import "$image_file" + image_count=$((image_count + 1)) + fi + done + if [ "$image_count" -eq 0 ]; then + echo "ERROR: no tar files found in /root/images" >&2 + exit 1 + fi + echo "imported $image_count images" + preKubeadmCommands: + - hostnamectl set-hostname "{{ ds.meta_data.hostname }}" + - echo "::1 ipv6-localhost ipv6-loopback localhost6 localhost6.localdomain6" >/etc/hosts + - echo "127.0.0.1 {{ ds.meta_data.hostname }} {{ local_hostname }} localhost localhost.localdomain localhost4 localhost4.localdomain4" >>/etc/hosts + - while ! ip route | grep -q "default via"; do sleep 1; done; echo "NetworkManager started" + - /usr/local/bin/capv-load-local-images.sh + postKubeadmCommands: + - chmod 600 /var/lib/kubelet/config.yaml + clusterConfiguration: + imageRepository: ${IMAGE_REGISTRY_ADDRESS}/tkestack + dns: + imageTag: ${DNS_IMAGE_TAG} + etcd: + local: + imageTag: ${ETCD_IMAGE_TAG} + apiServer: + extraArgs: + admission-control-config-file: /etc/kubernetes/admission/psa-config.yaml + audit-log-format: json + audit-log-maxage: "30" + audit-log-maxbackup: "10" + audit-log-maxsize: "200" + audit-log-mode: batch + audit-log-path: /etc/kubernetes/audit/audit.log + audit-policy-file: /etc/kubernetes/audit/policy.yaml + encryption-provider-config: /etc/kubernetes/encryption-provider.conf + kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt + profiling: "false" + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 + tls-min-version: VersionTLS12 + extraVolumes: + - hostPath: /etc/kubernetes + mountPath: /etc/kubernetes + name: vol-dir-0 + pathType: Directory + controllerManager: + extraArgs: + bind-address: "::" + cloud-provider: external + flex-volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" + profiling: "false" + tls-min-version: VersionTLS12 + scheduler: + extraArgs: + bind-address: "::" + profiling: "false" + tls-min-version: VersionTLS12 + initConfiguration: + nodeRegistration: + criSocket: /var/run/containerd/containerd.sock + ignorePreflightErrors: + - ImagePull + kubeletExtraArgs: + cloud-provider: external + node-labels: kube-ovn/role=master + volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" + name: '{{ local_hostname }}' + patches: + directory: /etc/kubernetes/patches + joinConfiguration: + nodeRegistration: + criSocket: /var/run/containerd/containerd.sock + ignorePreflightErrors: + - ImagePull + kubeletExtraArgs: + cloud-provider: external + node-labels: kube-ovn/role=master + volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" + name: '{{ local_hostname }}' + patches: + directory: /etc/kubernetes/patches +``` + +`wl-prod/30-infra-pool.yaml` — the `infra` slots. The manifest shows three nodes; take the count from +[Size the clusters](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#size-the-clusters) +and keep `replicas` in `32-infra.yaml` equal to it. + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: VSphereMachineConfigPool +metadata: + name: ${CLUSTER_NAME}-infra-pool + namespace: cpaas-system +spec: + clusterRef: + apiVersion: cluster.x-k8s.io/v1beta1 + kind: Cluster + name: ${CLUSTER_NAME} + datacenter: "${VSPHERE_DATACENTER}" + releaseDelayHours: 24 + configs: + - hostname: "${CLUSTER_NAME}-infra-1" + datacenter: "${VSPHERE_DATACENTER}" + network: + primary: + networkName: "${VSPHERE_NETWORK}" + ip: "${INFRA_NODE_IP_1}/${NODE_PREFIX}" + gateway: "${NODE_GATEWAY}" + dns: + - "${NODE_DNS}" + persistentDisks: + - name: var-cpaas + sizeGiB: ${INFRA_VAR_CPAAS_SIZE_GIB} + mountPath: /var/cpaas + fsFormat: ext4 + ephemeralDisks: + - name: var-lib-kubelet + sizeGiB: ${INFRA_VAR_LIB_KUBELET_SIZE_GIB} + mountPath: /var/lib/kubelet + fsFormat: ext4 + - name: var-lib-containerd + sizeGiB: ${INFRA_VAR_LIB_CONTAINERD_SIZE_GIB} + mountPath: /var/lib/containerd + fsFormat: ext4 + - hostname: "${CLUSTER_NAME}-infra-2" + datacenter: "${VSPHERE_DATACENTER}" + network: + primary: + networkName: "${VSPHERE_NETWORK}" + ip: "${INFRA_NODE_IP_2}/${NODE_PREFIX}" + gateway: "${NODE_GATEWAY}" + dns: + - "${NODE_DNS}" + persistentDisks: + - name: var-cpaas + sizeGiB: ${INFRA_VAR_CPAAS_SIZE_GIB} + mountPath: /var/cpaas + fsFormat: ext4 + ephemeralDisks: + - name: var-lib-kubelet + sizeGiB: ${INFRA_VAR_LIB_KUBELET_SIZE_GIB} + mountPath: /var/lib/kubelet + fsFormat: ext4 + - name: var-lib-containerd + sizeGiB: ${INFRA_VAR_LIB_CONTAINERD_SIZE_GIB} + mountPath: /var/lib/containerd + fsFormat: ext4 + - hostname: "${CLUSTER_NAME}-infra-3" + datacenter: "${VSPHERE_DATACENTER}" + network: + primary: + networkName: "${VSPHERE_NETWORK}" + ip: "${INFRA_NODE_IP_3}/${NODE_PREFIX}" + gateway: "${NODE_GATEWAY}" + dns: + - "${NODE_DNS}" + persistentDisks: + - name: var-cpaas + sizeGiB: ${INFRA_VAR_CPAAS_SIZE_GIB} + mountPath: /var/cpaas + fsFormat: ext4 + ephemeralDisks: + - name: var-lib-kubelet + sizeGiB: ${INFRA_VAR_LIB_KUBELET_SIZE_GIB} + mountPath: /var/lib/kubelet + fsFormat: ext4 + - name: var-lib-containerd + sizeGiB: ${INFRA_VAR_LIB_CONTAINERD_SIZE_GIB} + mountPath: /var/lib/containerd + fsFormat: ext4 +``` + +`wl-prod/31-infra-templates.yaml` — the `infra` VM template and bootstrap template. + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: VSphereMachineTemplate +metadata: + name: ${CLUSTER_NAME}-infra + namespace: cpaas-system +spec: + template: + spec: + server: "${VSPHERE_SERVER}" + template: "${VSPHERE_TEMPLATE}" + cloneMode: fullClone + folder: "${VSPHERE_FOLDER}" + datastore: "${VSPHERE_DATASTORE}" + diskGiB: ${INFRA_SYSTEM_DISK_GIB} + memoryMiB: ${INFRA_MEMORY_MIB} + numCPUs: ${INFRA_NUM_CPUS} + os: Linux + powerOffMode: trySoft + network: + devices: + - networkName: "${VSPHERE_NETWORK}" + machineConfigPoolRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: VSphereMachineConfigPool + name: ${CLUSTER_NAME}-infra-pool + namespace: cpaas-system +--- +apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 +kind: KubeadmConfigTemplate +metadata: + name: ${CLUSTER_NAME}-infra-bootstrap + namespace: cpaas-system +spec: + template: + spec: + files: + - path: /etc/resolv.conf + owner: "root:root" + append: false + permissions: "0644" + content: | + nameserver ${NODE_DNS} + - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json + owner: "root:root" + permissions: "0644" + content: | + { + "apiVersion": "kubelet.config.k8s.io/v1beta1", + "kind": "KubeletConfiguration", + "imagePullCredentialsVerificationPolicy": "NeverVerify", + "protectKernelDefaults": true, + "staticPodPath": null, + "streamingConnectionIdleTimeout": "5m", + "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt", + "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key" + } + - path: /usr/local/bin/capv-load-local-images.sh + owner: "root:root" + permissions: "0755" + content: | + #!/bin/bash + set -euo pipefail + until mountpoint -q /var/lib/containerd; do + echo "waiting for /var/lib/containerd mount" + sleep 1 + done + systemctl restart containerd + until systemctl is-active --quiet containerd; do + echo "waiting for containerd" + sleep 1 + done + if [ ! -d "/root/images" ]; then + echo "ERROR: /root/images directory not found" >&2 + exit 1 + fi + image_count=0 + for image_file in /root/images/*.tar; do + if [ -f "$image_file" ]; then + echo "importing image: $image_file" + ctr -n k8s.io images import "$image_file" + image_count=$((image_count + 1)) + fi + done + if [ "$image_count" -eq 0 ]; then + echo "ERROR: no tar files found in /root/images" >&2 + exit 1 + fi + echo "imported $image_count images" + joinConfiguration: + nodeRegistration: + criSocket: /var/run/containerd/containerd.sock + ignorePreflightErrors: + - ImagePull + kubeletExtraArgs: + cloud-provider: external + volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/" + name: '{{ local_hostname }}' + patches: + directory: /etc/kubernetes/patches + preKubeadmCommands: + - hostnamectl set-hostname "{{ ds.meta_data.hostname }}" + - echo "::1 ipv6-localhost ipv6-loopback localhost6 localhost6.localdomain6" >/etc/hosts + - echo "127.0.0.1 {{ ds.meta_data.hostname }} {{ local_hostname }} localhost localhost.localdomain localhost4 localhost4.localdomain4" >>/etc/hosts + - while ! ip route | grep -q "default via"; do sleep 1; done; echo "NetworkManager started" + - /usr/local/bin/capv-load-local-images.sh + postKubeadmCommands: + - chmod 600 /var/lib/kubelet/config.yaml + users: + - name: boot + sudo: ALL=(ALL) NOPASSWD:ALL + sshAuthorizedKeys: + - "${BOOT_SSH_PUBLIC_KEY}" +``` + +`wl-prod/32-infra.yaml` — the `infra` `MachineDeployment`. Labels and the taint are declared here, on the +pool, never by hand on nodes: `node-role.kubernetes.io/infra` is what the plugins select and tolerate, and +`node-role.kubernetes.io/worker` is what the default worker MachineConfigPool selects — without it these +nodes silently miss the machine configuration. + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: MachineDeployment +metadata: + name: ${CLUSTER_NAME}-infra + namespace: cpaas-system +spec: + clusterName: ${CLUSTER_NAME} + replicas: 3 # = number of slots in 30-infra-pool.yaml + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + maxUnavailable: 1 + selector: + matchLabels: + nodepool: infra + template: + metadata: + labels: + cluster.x-k8s.io/cluster-name: ${CLUSTER_NAME} + nodepool: infra + node-role.kubernetes.io/infra: "" + node-role.kubernetes.io/worker: "" + spec: + clusterName: ${CLUSTER_NAME} + version: "${K8S_VERSION}" + taints: + - key: node-role.kubernetes.io/infra + value: reserved + effect: NoSchedule + bootstrap: + configRef: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: ${CLUSTER_NAME}-infra-bootstrap + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: VSphereMachineTemplate + name: ${CLUSTER_NAME}-infra +``` + +#### Apply and verify + +Apply the files in order. Each step names its pass criteria; do not start the next step until they hold. + +**1. Cluster root.** + +```bash +# on global +apply_rendered wl-prod.env wl-prod/10-cluster.yaml "${GLOBAL_KUBECONFIG}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get cluster,vspherecluster "${CLUSTER_NAME}" +``` + +**2. CPI delivery.** + +```bash +# on global +apply_rendered wl-prod.env wl-prod/15-vsphere-cpi.yaml "${GLOBAL_KUBECONFIG}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get clusterresourceset "${CLUSTER_NAME}-vsphere-cpi" +``` + +**3. Control-plane slots.** + +```bash +# on global +apply_rendered wl-prod.env wl-prod/20-control-plane-pool.yaml "${GLOBAL_KUBECONFIG}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get vspheremachineconfigpool "${CLUSTER_NAME}-cp-pool" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get vspheremachineconfigpool "${CLUSTER_NAME}-cp-pool" \ + -o jsonpath='{range .status.conditions[*]}{.type}={.status} ({.reason}){"\n"}{end}' +``` + +Pass: `TOTAL` is `3`; `ClusterRefReady`, `VCenterAvailable`, `MembersValid`, `MembersUnique` and `Ready` +are `True`. `VCenterAvailable=False` means the identity Secret, the thumbprint or the reachability of +vCenter from `global` is wrong — fix it now, before any VM exists. + +**4. Control plane.** + +```bash +# on global +apply_rendered wl-prod.env wl-prod/21-control-plane-template.yaml "${GLOBAL_KUBECONFIG}" +apply_rendered wl-prod.env wl-prod/22-control-plane.yaml "${GLOBAL_KUBECONFIG}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get kubeadmcontrolplane "${CLUSTER_NAME}-kcp" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get machine \ + -l cluster.x-k8s.io/cluster-name="${CLUSTER_NAME}" -w +``` + +The three control-plane Machines come up one after the other. If one stalls, stop the watch (`-w` +accepts a single resource type) and read its infrastructure objects: +`kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get vspheremachine,vspherevm -l cluster.x-k8s.io/cluster-name="${CLUSTER_NAME}"`. +While the Machines come up, confirm the new cluster's +own `ClusterModule` version as soon as it exists +(`kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get clustermodule "${CLUSTER_NAME}"`) and, if it differs +from `global`'s, repeat the `alive` readiness check for the version it resolves to: the VIP mode can no +longer be changed once the control plane has bootstrapped. + +When the `KubeadmControlPlane` reports three ready replicas, fetch the workload kubeconfig as the +[kubeconfig convention](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#prepare-the-parameter-files) +describes, then check the endpoint, the CPI and the nodes: + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get vspherecluster "${CLUSTER_NAME}" \ + -o jsonpath='{range .status.conditions[*]}{.type}={.status} ({.reason}){"\n"}{end}' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get clusterresourcesetbinding "${CLUSTER_NAME}" +(umask 077; kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get secret "${CLUSTER_NAME}-kubeconfig" \ + -o jsonpath='{.data.value}' | base64 -d > "${CLUSTER_KUBECONFIG}") + +# on the workload cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get --raw=/version +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n kube-system get pods -l app=alive -o wide +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get apprelease alive +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n kube-system get daemonset vsphere-cloud-controller-manager +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes -o wide +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes \ + -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.providerID}{"\n"}{end}' +``` + +Pass: + +- `SelfBuiltLoadBalancerReady=True` and `KubeOvnAppReleaseReady=True` on the `VSphereCluster`. Neither + gates `VSphereCluster` readiness, so the infrastructure can report ready while the VIP still converges — + read the conditions, not only `READY`. +- `/version` answers through the VIP **with** TLS verification (the kubeconfig carries the cluster CA); + one `alive` Pod per control-plane node; the `alive` `AppRelease` is healthy. +- The CPI DaemonSet is ready on every node, and every node has a `vsphere://` `providerID`. A node + without one usually means a duplicate guest hostname in the vCenter or a CPI that never got its + configuration — see the product + [Troubleshooting](https://docs.alauda.io/immutable-infra/1.0/create-cluster/vmware-vsphere.html#troubleshooting). +- Three nodes are `Ready`, named `-cp-`, with `INTERNAL-IP` equal to their slot IPs. + +**5. `infra` slots** — same pass criteria as step 3, for `${CLUSTER_NAME}-infra-pool`. + +```bash +# on global +apply_rendered wl-prod.env wl-prod/30-infra-pool.yaml "${GLOBAL_KUBECONFIG}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get vspheremachineconfigpool "${CLUSTER_NAME}-infra-pool" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get vspheremachineconfigpool "${CLUSTER_NAME}-infra-pool" \ + -o jsonpath='{range .status.conditions[*]}{.type}={.status} ({.reason}){"\n"}{end}' +``` + +**6. `infra` pool.** + +```bash +# on global +apply_rendered wl-prod.env wl-prod/31-infra-templates.yaml "${GLOBAL_KUBECONFIG}" +apply_rendered wl-prod.env wl-prod/32-infra.yaml "${GLOBAL_KUBECONFIG}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get machinedeployment "${CLUSTER_NAME}-infra" -w +``` + +When the `MachineDeployment` reports three ready replicas, stop the watch and check the nodes and disks: + +```bash +# on the workload cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes -l node-role.kubernetes.io/infra -o wide +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes \ + -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key' + +# on global — persistent disks of both pools +for pool in cp-pool infra-pool; do + kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get vspheremachineconfigpool "${CLUSTER_NAME}-${pool}" \ + -o jsonpath='{range .status.persistentDiskStatuses[*]}{.hostname}{" "}{.name}{" "}{.phase}{"\n"}{end}' +done +``` + +Pass: the `MachineDeployment` has three ready replicas; the `infra` nodes are `Ready`, carry the +`node-role.kubernetes.io/infra` and `node-role.kubernetes.io/worker` labels **on the Node** (not only on +the Machine — a label only on the Machine means label sync is off on `global`) and the +`node-role.kubernetes.io/infra=reserved:NoSchedule` taint; every `var-cpaas` disk reports `Attached`. + +**Optional, before installing plugins: prove the slot model.** Delete one `infra` Machine +(`kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system delete machine `). Its replacement +must come back with the same hostname, the same IP and the same `var-cpaas` disk (`Attached` again), and +the slot returns to `InUse`. This is the cheapest moment to find a pool, datastore or template problem +that would otherwise surface during the first upgrade. + +Archive the parameter files and the manifests. By construction they contain no credential and no key; +confirm it in the delivery directory before archiving — the command must print nothing: + +```bash +# on the delivery host, in the delivery directory +grep -rnIiE 'aescbc|password[[:space:]]*[:=][[:space:]]*[^[:space:]]|_TOKEN=|BEGIN [A-Z ]*PRIVATE KEY' . +``` + +`CLUSTER_KUBECONFIG` is a cluster-admin credential: keep it outside the delivery directory and out of +the archive. + +### Variants + +#### Single cluster + +Does not apply: the single-cluster topology makes `global` the workload cluster, and `global` is never +on vSphere in this delivery. + +#### Dedicated pools + +The dedicated layout replaces the `infra` pool with `ingress`, `monitor` and `log`, each with its own +three files (`30`–`32` ingress, `33`–`35` monitor, `36`–`38` log). Each set is a copy of the `infra` set +with every `infra` replaced by the pool name — names, hostnames, selector, labels and taint key — and its +own node IPs, CPU, memory and `/var/cpaas` size, named like the `LOG_*` rows of the parameter table +(`INGRESS_*` and `MONITOR_*` likewise). The system disk and the ephemeral disks keep the `INFRA_*` +sizes. The `log` set: + +```diff +# wl-prod/36-log-pool.yaml (copy of 30-infra-pool.yaml), shown for the first slot; repeat per slot + metadata: +- name: ${CLUSTER_NAME}-infra-pool ++ name: ${CLUSTER_NAME}-log-pool + ... +- - hostname: "${CLUSTER_NAME}-infra-1" ++ - hostname: "${CLUSTER_NAME}-log-1" + ... +- ip: "${INFRA_NODE_IP_1}/${NODE_PREFIX}" ++ ip: "${LOG_NODE_IP_1}/${NODE_PREFIX}" + ... +- sizeGiB: ${INFRA_VAR_CPAAS_SIZE_GIB} ++ sizeGiB: ${LOG_VAR_CPAAS_SIZE_GIB} +``` + +```diff +# wl-prod/37-log-templates.yaml (copy of 31-infra-templates.yaml) + kind: VSphereMachineTemplate + metadata: +- name: ${CLUSTER_NAME}-infra ++ name: ${CLUSTER_NAME}-log + ... +- memoryMiB: ${INFRA_MEMORY_MIB} +- numCPUs: ${INFRA_NUM_CPUS} ++ memoryMiB: ${LOG_MEMORY_MIB} ++ numCPUs: ${LOG_NUM_CPUS} + ... +- name: ${CLUSTER_NAME}-infra-pool ++ name: ${CLUSTER_NAME}-log-pool + ... + kind: KubeadmConfigTemplate + metadata: +- name: ${CLUSTER_NAME}-infra-bootstrap ++ name: ${CLUSTER_NAME}-log-bootstrap +``` + +```diff +# wl-prod/38-log.yaml (copy of 32-infra.yaml) + kind: MachineDeployment + metadata: +- name: ${CLUSTER_NAME}-infra ++ name: ${CLUSTER_NAME}-log + ... + matchLabels: +- nodepool: infra ++ nodepool: log + ... +- nodepool: infra +- node-role.kubernetes.io/infra: "" ++ nodepool: log ++ node-role.kubernetes.io/log: "" + node-role.kubernetes.io/worker: "" + ... + taints: +- - key: node-role.kubernetes.io/infra ++ - key: node-role.kubernetes.io/log + value: reserved + effect: NoSchedule + ... +- name: ${CLUSTER_NAME}-infra-bootstrap ++ name: ${CLUSTER_NAME}-log-bootstrap + ... +- name: ${CLUSTER_NAME}-infra ++ name: ${CLUSTER_NAME}-log +``` + +Each dedicated pool has its **own** taint key, so a plugin with the wrong placement stays `Pending` +instead of silently landing on another pool; keep `node-role.kubernetes.io/worker` on every pool so +machine configuration reaches it. Apply and verify each set like steps 5 and 6, one pool at a time, and +extend the disk loop of step 6 to the new pool names. Plugin placement for this layout is in +[Install the platform plugins](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#install-the-platform-plugins). + +#### External load balancer instead of the Self-built VIP + +Use it when the customer mandates a load balancer device for API endpoints, or when the `alive` plugin +is not deployable. Remove the whole `controlPlaneLoadBalancer` block from `10-cluster.yaml` and drop +`CONTROL_PLANE_VRID` from the cluster file; `CONTROL_PLANE_VIP` becomes the load balancer's frontend +address: + +```diff + spec: + controlPlaneEndpoint: + host: "${CONTROL_PLANE_VIP}" + port: 6443 +- controlPlaneLoadBalancer: # write-once: decided at creation, never changed +- type: internal +- host: "${CONTROL_PLANE_VIP}" # must equal controlPlaneEndpoint.host +- port: 6443 # must equal controlPlaneEndpoint.port +- vrid: ${CONTROL_PLANE_VRID} + identityRef: +``` + +Provision the load balancer before step 1 per the +[External LoadBalancer Contract](https://docs.alauda.io/immutable-infra/1.0/how-to/control-plane-endpoint.html#external-loadbalancer-contract): +Layer 4 TCP with TLS passthrough, the three control-plane slot IPs as backends on `6443`, an HTTPS +`/healthz` health check, and a named owner who maintains the backends. Skip the `alive` check in +[Prepare the `global` cluster](#prepare-the-global-cluster), and in step 4 replace the Self-built VIP +checks with `/version` through the load balancer. The decision is as final as the Self-built VIP: a +cluster created this way can never move to it. + +#### Failure domains and a second NIC + +Both are creation-time variants of the product page: +[Multiple datacenters and failure domains](https://docs.alauda.io/immutable-infra/1.0/create-cluster/vmware-vsphere.html#multiple-datacenters-and-failure-domains) +and +[Creation-Time Topology Variants](https://docs.alauda.io/immutable-infra/1.0/create-cluster/vmware-vsphere.html#creation-time-topology-variants). +With a second NIC, also set `controlPlaneLoadBalancer.interface` (and `deviceName` in the slots) +explicitly. Add one variant at a time to this manifest set and validate it before combining them. + +#### Worker pools + +Application workers follow the customer's machines and are not part of this delivery. When the cluster +needs a generic pool, copy the three `infra` files to `40`–`42` with `worker` in place of `infra` in the +names, hostnames, selector and variables, then remove the `node-role.kubernetes.io/infra` label and the +taint from the `MachineDeployment` so that ordinary workloads can schedule there. Keep +`node-role.kubernetes.io/worker`, the standard disk set, `maxSurge: 0` with `maxUnavailable: 1`, and +`replicas` no larger than the number of slots in the pool. + +### Continue with the common steps + +With the cluster and its pools up, continue in the common article on the new workload cluster: + +1. [Install the platform plugins](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#install-the-platform-plugins) + — placement on `infra` (or the dedicated pools). `INGRESS_VIP_RANGE` (and the `INGRESS_VIP` in it) must lie in + the node subnet of `VSPHERE_NETWORK`, outside every slot IP and `CONTROL_PLANE_VIP`; in Layer 2 mode + MetalLB announces it with ARP, so the port-group policy that lets the Self-built VIP's gratuitous ARP + through must allow it as well. +2. [Configure machines](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#configure-machines) + — the Machine Configuration plugin on this cluster first, then time zone, NTP and the `boot` SSH key. +3. [Accept the delivery](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md#accept-the-delivery) + — plus the vSphere-specific checks below. + +| Check | Evidence | Pass | +| --- | --- | --- | +| Slot pools | Commands below | Every pool of the cluster `Ready=True`; `ALLOCATED` equals its running Machines | +| Persistent disks | Loop in step 6 | Every `var-cpaas` disk `Attached` | +| Ephemeral disks | Commands below | Every ephemeral disk of every allocated slot listed with a unit number | +| Node identity | `providerID` command in step 4 | Every node has a `vsphere://` ID; names and IPs equal the slots | +| CPI | `ClusterResourceSetBinding` and DaemonSet in step 4 | Binding exists; DaemonSet ready on every node | +| Endpoint | `VSphereCluster` conditions and `/version` in step 4 | `SelfBuiltLoadBalancerReady=True`; one `alive` Pod per control-plane node; `/version` with TLS verification | +| Admission | Webhook command in [Prepare the `global` cluster](#prepare-the-global-cluster) | Both `capv` rollout webhooks registered | +| Rollout settings | Commands below | `maxSurge: 0` on the control plane and every pool; `maxUnavailable: 1` on every `MachineDeployment` | +| Archive | `grep` before archiving | No credential, key or kubeconfig in the archive | + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get vspheremachineconfigpool +for pool in cp-pool infra-pool; do + kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get vspheremachineconfigpool "${CLUSTER_NAME}-${pool}" \ + -o jsonpath='{range .status.ephemeralDiskStatuses[*]}{.hostname}{" "}{.name}{" unit="}{.unitNumber}{"\n"}{end}' +done +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get kubeadmcontrolplane "${CLUSTER_NAME}-kcp" \ + -o jsonpath='{.spec.replicas}{" "}{.spec.rolloutStrategy.rollingUpdate}{"\n"}' +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get machinedeployment -l cluster.x-k8s.io/cluster-name="${CLUSTER_NAME}" \ + -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.strategy.rollingUpdate}{"\n"}{end}' +``` + +Day-2 operations — scale-out (add slots first, then raise `replicas`), template changes, Kubernetes +upgrades and provider upgrades — follow +[Managing Nodes on VMware vSphere](https://docs.alauda.io/immutable-infra/1.0/manage-nodes/vmware-vsphere.html), +[Upgrading Clusters on VMware vSphere](https://docs.alauda.io/immutable-infra/1.0/upgrade-cluster/vmware-vsphere.html) +and [Upgrading the Provider](https://docs.alauda.io/immutable-infra/1.0/install/vmware-vsphere.html#upgrade-the-provider). +Re-applying an archived `MachineDeployment` is accepted only while it still carries `maxSurge: 0` and +`maxUnavailable: 1`; the control-plane manifest is never re-applied. + +If the delivery has to deviate from this design, stop and resolve the deviation with the platform team +before continuing — on vSphere most of these decisions are frozen at creation, so a deviation found later +is a rebuild. + +## Related Information + +- [How to Deliver ACP with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md) — the common article +- [Creating Clusters on VMware vSphere](https://docs.alauda.io/immutable-infra/1.0/create-cluster/vmware-vsphere.html) +- [VMware vSphere Infrastructure Preparation](https://docs.alauda.io/immutable-infra/1.0/infrastructure/vmware-vsphere.html) +- [VMware vSphere Provider Installation](https://docs.alauda.io/immutable-infra/1.0/install/vmware-vsphere.html) +- [VMware vSphere Provider Release Notes](https://docs.alauda.io/immutable-infra/1.0/release-notes/vmware_vsphere.html#v1-0-17) +- [Plan the Control Plane Endpoint](https://docs.alauda.io/immutable-infra/1.0/how-to/control-plane-endpoint.html) +- [Configure Annotations, Labels, and Taints on Nodes](https://docs.alauda.io/immutable-infra/1.0/how-to/node-configuration.html) +- [Managing Nodes on VMware vSphere](https://docs.alauda.io/immutable-infra/1.0/manage-nodes/vmware-vsphere.html) +- [Upload Packages](https://docs.alauda.io/container_platform/4.4/extend/upload_package.html#violet_push_usage) and [Cluster Plugin](https://docs.alauda.io/container_platform/4.4/extend/cluster_plugin.html) +- [Evaluating Resources for Workload Cluster](https://docs.alauda.io/container_platform/4.4/configure/scalability/evaluating_workload.html) diff --git a/docs/en/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md b/docs/en/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md new file mode 100644 index 000000000..14e900925 --- /dev/null +++ b/docs/en/solutions/acp/How_to_Deliver_ACP_with_Immutable_Infrastructure.md @@ -0,0 +1,2450 @@ +--- +kind: + - How To +products: + - Alauda Container Platform +ProductsVersion: + - 4.4.x +--- + +# How to Deliver ACP with Immutable Infrastructure + +## Purpose + +This guide describes the standard delivery of Alauda Container Platform (ACP) 4.4 +on Immutable Infrastructure — clusters whose nodes run Alauda OS and are managed +through Cluster API. It is the provider-neutral part of a set of five articles: the +reference design, the parameter files every manifest is rendered from, the +end-to-end delivery flow, the platform plugins, machine configuration, sizing, and +acceptance. Four provider articles add what differs per infrastructure: the +provider design, the `global` installation with disaster recovery (DR), and +workload cluster creation. + +The product documentation remains the field-level reference. These articles fix +the choices — which option, which value, in which order — assemble them into +complete manifests with `${VAR}` placeholders, and give the check that proves each +step worked. + +Scope and assumptions: + +- **ACP 4.4, Alauda OS nodes only.** For a non-Alauda OS, use the + [standard installation path](https://docs.alauda.io/container_platform/4.4/install/index.html). +- **Two supported topologies.** A dedicated `global` cluster (primary + standby + for DR) managing N workload clusters is the main line; a single cluster in which + `global` doubles as the workload cluster is a supported variant. +- **No commercial load balancer is assumed** for the platform and cluster API + VIPs. A customer's commercial load balancer, if any, serves application ingress + only. +- **Worker pools are sized by the customer.** They follow the customer's + machines; these articles cover the clusters and the platform node pools, but + every cluster needs worker pools before handover (see + [Node pools, labels and taints](#node-pools-labels-and-taints)). +- **Application storage is out of scope** and is planned separately. + +**Validation status.** The manifests are assembled from the ACP 4.4 product +documentation and the provider versions named in the provider articles. Validate +them in a staging environment before a production delivery. + +## Resolution + +### How to use these articles + +A delivery chooses a provider twice: once for `global` and once for the workload +clusters. They can differ — for example `global` on Huawei DCS and workload +clusters on VMware vSphere. + +1. Read this article's [Reference design](#reference-design), + [Prepare the parameter files](#prepare-the-parameter-files) and + [Delivery flow](#delivery-flow) end to end. +2. Open the provider article for the infrastructure that hosts `global`; install + `global` and add disaster recovery. +3. Open the provider article for the workload-cluster infrastructure and create + the workload clusters. +4. Return here for [Install the platform plugins](#install-the-platform-plugins), + [Configure machines](#configure-machines) and + [Accept the delivery](#accept-the-delivery) on every cluster. + +| Provider | `global` (primary + standby) | Workload clusters | Article | +| --- | --- | --- | --- | +| Huawei DCS | Yes | Yes | [How to Deliver ACP on Huawei DCS with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_on_Huawei_DCS_with_Immutable_Infrastructure.md) | +| Huawei Cloud Stack | Yes | Yes | [How to Deliver ACP on Huawei Cloud Stack with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_on_Huawei_Cloud_Stack_with_Immutable_Infrastructure.md) | +| Bare Metal | Yes, including preparing `global` as the management cluster | Yes | [How to Deliver ACP on Bare Metal with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_on_Bare_Metal_with_Immutable_Infrastructure.md) | +| VMware vSphere | No — `global` runs on another provider and already exists | Yes | [How to Deliver ACP on VMware vSphere with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_on_VMware_vSphere_with_Immutable_Infrastructure.md) | + +Conventions shared by all five articles: + +- Every command block starts with a comment that names the cluster it runs + against — `# on the bootstrap cluster`, `# on global`, + `# on the workload cluster`, or `# on the cluster` for a step that applies to + every cluster, `global` included — or the host it runs on (`# on the delivery + host`, `# on a node`). See [Kubeconfig convention](#kubeconfig-convention). +- Manifests are shown as files, `/-.yaml`, numbered in + apply order. Provider articles number the cluster manifests below `50`; this + article uses `50` to `79`. Save each block as plain YAML under that name. +- `${VAR}` placeholders are filled from the parameter files by the + [`render` helper](#the-render-helper); nothing else in a manifest is edited by + hand. +- Every command runs in Bash, from the delivery directory; see + [The render helper](#the-render-helper). + +### Reference design + +#### Component selection + +Only choices with a real alternative are listed; everything else follows the +product defaults. + +| Area | Selected | Alternative | Why | +| --- | --- | --- | --- | +| Monitoring | **Monitoring for VictoriaMetrics** | Monitoring for Prometheus | Highly available installation and lower resource usage per the [component selection guide](https://docs.alauda.io/container_platform/4.4/observability/monitor/architecture/component_selection_suggestion.html) | +| Log storage | **ClickHouse** | OpenSearch | Lowest resource usage per ingest unit and the highest throughput per node ([Log Component Selection Guide](https://docs.alauda.io/logging-service/4.4/architecture/component_suggestion.html)); the sizing method below is built on the ClickHouse profiles. ClickHouse does not support IPv6-only or dual-stack workload clusters — use OpenSearch there | +| Log buffer | **Kafka** (KRaft mode) | — | Mandatory on ACP 4.4 for both log backends: it buffers records between the collector and the storage backend | +| Application ingress | **Alauda Build of Ingress NGINX Controller** | Envoy Gateway | Current standard path | +| Application ingress VIP | **MetalLB** (Layer 2) | Customer's commercial load balancer | Mutually exclusive on the same VIP — an address cannot be announced by MetalLB (ARP) and forwarded by a load balancer device at the same time. With a commercial load balancer, set `IngressNginx.spec.controller.hostNetwork: true`; otherwise the controller's Service is `ClusterIP` and the load balancer has nothing to forward to | +| Cluster API VIP | Provider mechanism, see [Endpoints and addresses](#endpoints-and-addresses) | External load balancer | No commercial load balancer is assumed | + +#### Endpoints and addresses + +The **cluster endpoint is always an IP address (a VIP), never a domain name.** The +API server certificate can only be issued by the cluster's own CA, so a +customer-issued certificate cannot be served on the endpoint, and a domain without +that certificate defeats its usual purpose. Only the **platform access address** +(`PLATFORM_HOST`) uses a domain, and only that domain switches during a DR +failover. Plan every VIP as a permanent address: it is written into the API server +certificate, every kubeconfig and every management connection, and changing it +later is a certificate and connectivity migration +([Plan the Control Plane Endpoint](https://docs.alauda.io/immutable-infra/1.0/how-to/control-plane-endpoint.html)). + +| Provider | Cluster API VIP | Condition | +| --- | --- | --- | +| Huawei DCS | `type: internal` Self-built VIP | DCS Provider `v1.0.22+` with ACP 4.4+. An external load balancer is the alternative | +| Huawei Cloud Stack | ELB created by the provider | — | +| VMware vSphere | `type: internal` Self-built VIP | vSphere Provider `v1.0.17+` with ACP 4.4+. A creation-time choice: it cannot be added to or removed from an existing cluster | +| Bare Metal | `type: Internal` (Alive) or `External` | — | + +A Self-built VIP needs an unused IPv4 address in the same Layer-2 network as the +control-plane interfaces, a VRID that is unique in that Layer-2 domain, and VRRP +and gratuitous ARP allowed between the control-plane nodes and the network. + +The **application ingress VIP** (`INGRESS_VIP`) comes from a MetalLB address pool +(`INGRESS_VIP_RANGE`). It must be Layer-2 adjacent to the nodes that announce it, +unused by the physical network (no gateway addresses), and outside every cluster +CIDR ([MetalLB constraints](https://docs.alauda.io/container_platform/4.4/networking/operators/metallb_operator.html)). + +#### Topologies + +**Multi-cluster** — one `global` pair (primary + standby) plus N workload clusters: + +``` +platform domain ──▶ global primary ──── etcd data ────▶ global standby + │ (runs etcd-sync) + │ manages + ▼ + workload cluster × N +``` + +- Each side of the `global` pair is a complete installation from its own + bootstrap node, with its own registry, monitoring and logging. +- The DR increments must be identical on both sides: the encryption provider + configuration created from one shared file, the same etcd server certificate + SANs, and the provider-specific DR fields. A mismatch installs cleanly and + surfaces only at the first failover. +- The `global` naming convention is a hard requirement: the CAPI `Cluster` and + the provider infrastructure cluster are named exactly `global`, and every other + resource of the `global` cluster carries the `global-` prefix. DR identifies the + resources of the `global` pair by that prefix; a resource without it is lost at + failover + ([naming convention](https://docs.alauda.io/immutable-infra/1.0/global/install.html#common-prerequisites)). + +**Single cluster** — `global` doubles as the workload cluster: one bootstrap node, +no DR, no cross-cluster management. `global` then also carries the application +ingress. A customer may run several such clusters that are deliberately +independent of each other. + +#### Node pools, labels and taints + +| Layout | Pools | When | +| --- | --- | --- | +| Consolidated | `control-plane` + one `infra` pool carrying ingress, monitoring and logging | Up to roughly 20 workers | +| Dedicated | `control-plane` + `ingress` + `monitor` + `log` | Larger clusters; recalculate sizing against the product capacity pages | + +`global` uses the consolidated layout. Worker pools for applications are added +per the customer's machines and carry no platform taint. The main line creates +only the control plane, which carries the kubeadm control-plane taint, and tainted +platform pools, so a new cluster has no untainted node. Application workloads, and +any component that does not tolerate those taints, need an untainted worker pool: +add the customer's worker pools as the provider article describes before handing +a cluster over. + +| Pool | Labels | Taint | +| --- | --- | --- | +| `infra` | `node-role.kubernetes.io/infra: ""`, `node-role.kubernetes.io/worker: ""` | `node-role.kubernetes.io/infra=reserved:NoSchedule` | +| `ingress` | `node-role.kubernetes.io/ingress: ""`, `node-role.kubernetes.io/worker: ""` | `node-role.kubernetes.io/ingress=reserved:NoSchedule` | +| `monitor` | `node-role.kubernetes.io/monitor: ""`, `node-role.kubernetes.io/worker: ""` | `node-role.kubernetes.io/monitor=reserved:NoSchedule` | +| `log` | `node-role.kubernetes.io/log: ""`, `node-role.kubernetes.io/worker: ""` | `node-role.kubernetes.io/log=reserved:NoSchedule` | + +Rules that make the layout work: + +- **Labels and taints are declared on the pool resources**, never applied by hand + on nodes: an Alauda OS node is replaced, not mutated, so a hand-made label + disappears with the next replacement + ([Cluster Node Planning](https://docs.alauda.io/container_platform/4.4/configure/nodes/node_planning.html#creating-infra-nodes-on-immutable-clusters)). + Taints are set on the machine template and applied at node registration. +- **A `Machine` label reaches its `Node` only after label synchronization is + enabled** on `capi-controller-manager` + ([Configure Annotations, Labels, and Taints on Nodes](https://docs.alauda.io/immutable-infra/1.0/how-to/node-configuration.html)). + Without it every Pod that selects a role label stays `Pending`, which looks + exactly like a wrong label. The delivery standard syncs the role labels with + `--additional-sync-machine-labels=node-role.kubernetes.io/.*` (the flag accepts + regular expressions). +- **Each dedicated pool has its own taint key.** A shared key combined with the + common `tolerations: [{operator: Exists}]` pattern silently opens every + dedicated pool to every plugin; distinct keys make a misplaced plugin `Pending` + immediately, which is the failure you want. +- **Every dedicated pool also carries `node-role.kubernetes.io/worker`.** The + default `worker` MachineConfigPool selects on it; without it the nodes silently + miss machine configuration (time zone, NTP, SSH key). +- **Use `NoSchedule`, not `NoExecute`**: a mislabeled pool during delivery should + block scheduling, not evict running Pods. + +Placement by cluster and layout (the pool each plugin runs on): + +| Cluster · layout | Ingress (MetalLB + Ingress NGINX) | Monitoring | Logging (Kafka, ClickHouse, log storage) | +| --- | --- | --- | --- | +| `global` · multi-cluster | — | `infra` | `infra` | +| `global` · single cluster | `infra` | `infra` | `infra` | +| Workload · consolidated | `infra` | `infra` | `infra` | +| Workload · dedicated pools | `ingress` | `monitor` | `log` | + +DaemonSets that must cover every node (log collector, node exporter, CNI, CSI, +machine configuration daemon) are not placed; they are checked for coverage in +[DaemonSet coverage](#daemonset-coverage). + +#### Disks + +**One persistent `/var/cpaas` disk per node.** Monitoring and the log collector +write to sub-directories under it; Alauda OS keeps node-local platform data under +`/var/cpaas`, and the root filesystem is read-only, so the plugin defaults under +`/cpaas/...` are replaced by `/var/cpaas/...` everywhere in this article. Declare +the disk with the provider's persistent-disk mechanism so that it survives node +replacement: + +| Provider | Persistent disk field | +| --- | --- | +| Huawei DCS | `DCSIpHostnamePool.spec.pool[].persistentDisk` | +| Huawei Cloud Stack | `HCSMachineConfigPool.spec.configs[].persistentDisks[]` | +| VMware vSphere | `VSphereMachineConfigPool.spec.configs[].persistentDisks[]` | +| Bare Metal | `MachineInventory.spec.storage.volumes[]` | + +| Path | Written by | Note | +| --- | --- | --- | +| `/var/cpaas/monitoring` | VictoriaMetrics `LocalVolume` storage | Set in the plugin installation; plan it as fixed | +| `/var/cpaas/data/nevermore` | Log collector working data | Cannot be changed after installation | + +Kafka, Keeper and ClickHouse store their data on PersistentVolumes of +`KAFKA_STORAGE_CLASS` and `CLICKHOUSE_STORAGE_CLASS`, not on a fixed node path. +Choose StorageClasses whose volumes survive node replacement; a node-local class +must provision under `/var/cpaas` (for example `/var/cpaas/data/clickhouse`) on +the persistent disk. + +Capacity is the sum of the uses: a consolidated `infra` node carries platform +state, log storage and monitoring data on one disk (see +[Size the clusters](#size-the-clusters)). The trade-off is isolation — logs filling +up can take the platform state with them — so compensate with log retention +limits and capacity alerts, not only with a bigger disk. + +Pools with persistent disks roll out with `maxSurge: 0` where the provider +requires it: a surge machine cannot take over the hostname, IP and disk slot of +the machine it replaces. The field paths differ — `KubeadmControlPlane` uses +`spec.rolloutStrategy.rollingUpdate.maxSurge` and has no `maxUnavailable`; +`MachineDeployment` uses `spec.strategy.rollingUpdate`. Copying the +`MachineDeployment` block onto a `KubeadmControlPlane` leaves the control plane at +the default surge. The provider article states which pools need it. + +#### Observability + +Every cluster observes itself: each cluster runs the full monitoring stack and its +own log storage, and each side of the `global` pair collects, stores and queries +only its own data. DR replicates only etcd and excludes `cpaas-system`, so +observability data is never copied to the standby +([Observability Placement](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#observability-placement)). +**Workload cluster metrics and logs are never written to either `global` +cluster** — that data becomes unavailable when the `global` roles change. + +On ACP 4.4 the log stack has five parts: on every storage cluster Kafka, +ClickHouse and Alauda Container Platform Log Storage Management (which derives +the ingest and query tier from a `PlatformLogForward`); on every collected cluster +the Log Collector; and on `global` only Log Essentials, which publishes the query +APIs used by the console. + +If the customer needs one query point across workload clusters, the product's +aggregation model applies instead: a dedicated workload cluster runs the +VictoriaMetrics Center and the log storage, and the other workload clusters run +agents that point at it — never at `global`. The manifests below implement the +per-cluster model; the aggregation model changes only the `crossClusterDependency` +values and the agent-only setting. + +#### Network flows and ports + +Freeze the port approvals before creating any cluster. Use the authoritative +pages instead of copying port tables into the delivery record: + +- Component ports: [Communication Matrix](https://docs.alauda.io/container_platform/4.4/install/prepare/communication_matrix.html), + including [What This Matrix Does Not Cover](https://docs.alauda.io/container_platform/4.4/install/prepare/communication_matrix.html#not_covered). + The host firewall of Alauda OS nodes is maintained by the platform; use the + matrix for the network firewalls between zones. +- `global` DR pair: the bidirectional [DR network requirements](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#network-requirements). +- Bootstrap node and provider API flows: the provider article and the + [bootstrap network requirements](https://docs.alauda.io/immutable-infra/1.0/global/install.html#bootstrap-host-network). + +### Prepare the parameter files + +#### Files and layout + +Every manifest of a delivery is rendered from two parameter files: `site.env` +(values common to every cluster of the delivery) and one file per cluster. Every +cluster file uses **the same variable names** — a new cluster file is a copy of an +existing one with different values. Combined with one manifest directory per +cluster, this makes it impossible to apply one cluster's values to another by +accident. + +``` +acp-delivery/ +├── site.env +├── global-primary.env +├── global-standby.env +├── wl-prod.env # one file per workload cluster: wl-.env +├── global-primary/ # manifests of the primary global (provider: <50, common: 50–79) +├── global-standby/ +└── wl-prod/ +``` + +Rules for the `.env` files: + +- One `NAME=value` per line, no `export`, single quotes around any value with + spaces. Use Unix line endings. +- **No credentials.** Passwords, tokens and keys never go into a `.env` file, a + manifest or the delivery record; see [Credentials](#credentials). +- Never deliver an example value unchanged. Provider articles add their own + variables to the same files; they never rename a variable defined here. + +#### The render helper + +Every command in these articles runs in **Bash**, the shell the product requires on +the bootstrap node. The helpers use Bash-only syntax (`${!name}` indirection, word +splitting of an unquoted variable, `read -s -p`); under zsh `render` silently +substitutes nothing. Save the block below as `acp-helpers.sh` in the delivery +directory and load it in every new shell with `. ./acp-helpers.sh`; it refuses to +load in another shell. + +`render` substitutes only the variables defined in `site.env` and the cluster +file; any other `$` in a manifest, such as a shell variable in an embedded script, +stays literal. It uses `envsubst` from gettext and falls back to pure Bash when +`envsubst` is missing. + +```bash +#!/usr/bin/env bash +# acp-helpers.sh — load with: . ./acp-helpers.sh (Bash only) +[ -n "${BASH_VERSION:-}" ] || { echo 'acp-helpers.sh: run these articles in bash' >&2; return 1 2>/dev/null || exit 1; } + +# Usage: render (writes the rendered manifest to stdout) +render() { + local vars + vars=$(sed -n 's/^\([A-Z][A-Z0-9_]*\)=.*/\1/p' site.env "$1" | sort -u) + ( + set -a; . ./site.env; . "./$1"; set +a + if command -v envsubst >/dev/null 2>&1; then + envsubst "$(printf '${%s} ' $vars)" < "$2" + else + local line v + while IFS= read -r line || [ -n "$line" ]; do + for v in $vars; do line=${line//"\${$v}"/"${!v}"}; done + printf '%s\n' "$line" + done < "$2" + fi + ) +} + +# Usage: load_env (loads site.env and one cluster file into this shell) +load_env() { set -a; . ./site.env; . "./$1"; set +a; } + +# Usage: apply_rendered +# Renders, refuses to continue on an unreplaced placeholder, validates server-side, then applies. +apply_rendered() { + render "$1" "$2" > /tmp/out.yaml || return 1 + if grep -nE '\$\{[A-Za-z_][A-Za-z0-9_]*\}' /tmp/out.yaml; then echo "unreplaced variables in $2" >&2; return 1; fi + kubectl --kubeconfig "$3" apply --dry-run=server -f /tmp/out.yaml >/dev/null || return 1 + kubectl --kubeconfig "$3" apply -f /tmp/out.yaml +} +``` + +The basic pattern, which `apply_rendered` wraps: + +```bash +render global-primary.env global-primary/10-control-plane.yaml > /tmp/out.yaml +grep -nE '\$\{[A-Za-z_][A-Za-z0-9_]*\}' /tmp/out.yaml && echo "unreplaced variables" || kubectl apply -f /tmp/out.yaml +``` + +A leftover `${NAME}` means a variable is missing from the parameter files. An +article states explicitly when a manifest contains an intentional `${...}`; the +Keeper manifest's shell expansion `${HOSTNAME##*-}` is not a placeholder, and +`apply_rendered` lets it pass. + +#### Credentials + +Credentials never appear in a `.env` file, a manifest, a command typed into shell +history, or the delivery record. Read them at a prompt, use them once, and forget +them; create Secrets with `kubectl create` and reference them by name. Pass +each value through process substitution, never `--from-literal`: a literal is part +of the `kubectl` command line and visible in the process list. + +```bash +# on the cluster that needs the Secret (the provider article or the step names the Secret and its keys) +read -rsp 'Password: ' SECRET_VALUE; echo +kubectl -n cpaas-system create secret generic \ + --from-file==<(printf '%s' "${SECRET_VALUE}") +unset SECRET_VALUE +``` + +Never `kubectl apply` a Secret manifest that contains a credential: `apply` +records the full object, credential included, in its +`last-applied-configuration` annotation. Keep shell tracing (`set -x`) off while a +credential is in the environment. The manifests in these articles contain no +credentials, so rendered files can be archived as they are. + +#### Kubeconfig convention + +Each cluster file names two kubeconfigs: + +- `CLUSTER_KUBECONFIG` — the cluster itself. Commands marked + `# on the workload cluster` (or `# on the cluster` for steps that apply to every + cluster) use it. +- `GLOBAL_KUBECONFIG` — the `global` cluster that manages this cluster. + `ModuleInfo` objects are always created there. In `global-primary.env` and + `global-standby.env` both variables point at that side itself, because each + side installs its own plugins. + +Kubeconfigs are credentials: keep them outside the delivery directory with mode +`0600`, copy them between hosts only over an encrypted channel, and do not +archive them. Cluster API stores each cluster's admin kubeconfig in the Secret +`-kubeconfig` on the cluster that manages it: + +```bash +umask 077; mkdir -p /root/.kube/acp +# on the bootstrap node of each side: the installation guide (Step 2) copies +# /var/cpaas/data/alauda.kubeconfig to /root/.kube/config — the provider articles +# call that path BOOTSTRAP_KUBECONFIG +# on the bootstrap cluster: the global it installed — read it before the bootstrap cluster is removed +load_env global-primary.env +kubectl --kubeconfig /root/.kube/config -n cpaas-system \ + get secret global-kubeconfig -o jsonpath='{.data.value}' | base64 -d > "${CLUSTER_KUBECONFIG}" +# on global: a workload cluster, after it is created +load_env wl-prod.env +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system \ + get secret "${CLUSTER_NAME}-kubeconfig" -o jsonpath='{.data.value}' | base64 -d > "${CLUSTER_KUBECONFIG}" +``` + +#### Common variables + +Provider articles reuse these names and add their own (for example `HOST_IP`, +`NODE_REGISTRY_ADDRESS`, `PROVIDER_SECRET_NAME`, and the DR variables). + +**`site.env`** — one value for the whole delivery: + +| Variable | Meaning | Example | Where to get it | +| --- | --- | --- | --- | +| `ACP_VERSION` | ACP distribution version of the Core Package | `v4.4.0` | The Core Package you download | +| `PLATFORM_HOST` | Platform access address; a domain that resolves to the active `global` VIP | `acp.example.com` | Customer DNS plan | +| `REGISTRY_DOMAIN` | Platform registry address, `:11443` | `acp.example.com:11443` | Provider article (`global` installation, Step 1) | +| `K8S_VERSION` | Kubernetes version, v-prefixed, matching the Alauda OS image | `v1.35.` | The Alauda OS image or VM template of the release (provider article) | +| `TIMEZONE` | Node time zone, tz database name | `Asia/Singapore` | Agreed with the customer | +| `NTP_SERVERS` | Customer's internal NTP servers, space-separated | `192.0.2.123 192.0.2.124` | Customer network plan; never a public NTP source in production | +| `BOOT_SSH_PUBLIC_KEY` | Public key for the `boot` user on every node | `ssh-ed25519 AAAAC3...Example delivery@example.com` | Generated by the delivery team; the private key stays with the customer's administrators | +| `CHRONY_CONF_B64` | Base64 of the generated `chrony.conf` | (generated) | Written by the command in [Set NTP](#set-ntp); do not edit | +| `MACHINE_CONFIG_VERSION` | Machine Configuration plugin version | `` | [Look up plugin versions](#look-up-plugin-versions) | +| `METALLB_MODULE_NAME` | Module name of the MetalLB cluster plugin | `` | [Look up plugin versions](#look-up-plugin-versions) | +| `METALLB_VERSION` | MetalLB plugin version | `` | [Look up plugin versions](#look-up-plugin-versions) | +| `VICTORIAMETRICS_VERSION` | VictoriaMetrics plugin version | `` | [Look up plugin versions](#look-up-plugin-versions) | +| `LOGAGENT_VERSION` | Log Collector plugin version | `` | [Look up plugin versions](#look-up-plugin-versions) | +| `LOG_API_VERSION` | Log Essentials plugin version | `` | [Look up plugin versions](#look-up-plugin-versions) | +| `INGRESS_NGINX_CHANNEL` | Update channel of the Ingress NGINX operator | `stable` | [Look up plugin versions](#look-up-plugin-versions) | +| `INGRESS_NGINX_CSV` | Operator version to install (`startingCSV`) | `` | [Look up plugin versions](#look-up-plugin-versions) | +| `CLICKHOUSE_OPERATOR_CSV` | ClickHouse operator version to install (`startingCSV`) | `` | [Look up plugin versions](#look-up-plugin-versions) | + +**Cluster files** (`global-primary.env`, `global-standby.env`, `wl-.env`) — +same names, one value per cluster: + +| Variable | Meaning | Example | Where to get it | +| --- | --- | --- | --- | +| `CLUSTER_NAME` | Cluster name; exactly `global` on both sides of the `global` pair | `global`, `wl-prod` | Naming plan | +| `CONTROL_PLANE_VIP` | Cluster API VIP — an IP address, never a domain | `192.0.2.10` (primary), `203.0.113.10` (standby), `192.0.2.20` (`wl-prod`) | Network plan | +| `CLUSTER_CIDR` | Pod CIDR | `100.3.0.0/16` (`global`), `198.18.0.0/16` (`wl-prod`) | Network plan. The Pod, Service and join CIDRs of every cluster must not overlap each other, another cluster's, or any node, VIP or customer network | +| `SERVICE_CIDR` | Service CIDR | `100.4.0.0/16` (`global`), `198.19.0.0/16` (`wl-prod`) | Network plan | +| `KUBE_OVN_JOIN_CIDR` | Kube-OVN join CIDR, a `/16` | `100.5.0.0/16` (`global`), `100.65.0.0/16` (`wl-prod`) | Network plan | +| `INGRESS_VIP_RANGE` | MetalLB address pool for the application ingress VIP; empty on a `global` without application ingress | `192.0.2.50/32` | Network plan | +| `INGRESS_VIP` | The ingress VIP itself, inside `INGRESS_VIP_RANGE` | `192.0.2.50` | Network plan; customer DNS points application domains at it | +| `GLOBAL_KUBECONFIG` | Kubeconfig of the `global` that manages this cluster; on a `global` side, that side itself | `/root/.kube/acp/global-primary.kubeconfig` | [Kubeconfig convention](#kubeconfig-convention) | +| `CLUSTER_KUBECONFIG` | Kubeconfig of this cluster | `/root/.kube/acp/wl-prod.kubeconfig` | [Kubeconfig convention](#kubeconfig-convention) | +| `MONITOR_SIZE` | VictoriaMetrics scale: `Small`, `Medium` or `Large` | `Medium` | [Size the clusters](#size-the-clusters) | +| `MONITOR_STORAGE_GI` | Monitoring storage per node, in Gi | `100` | [Size the clusters](#size-the-clusters) | +| `MONITOR_RETENTION_DAYS` | Metrics retention in days | `7` | Customer requirement | +| `MONITOR_STORAGE_NODES` | Node names that hold monitoring data, as a YAML list; one vmstorage replica per node | `'["wl-prod-infra-1", "wl-prod-infra-2", "wl-prod-infra-3"]'` | `kubectl get nodes -l node-role.kubernetes.io/infra` (or `.../monitor`) after the pool is up | +| `KAFKA_STORAGE_CLASS` | StorageClass for the Kafka broker and controller volumes | `` | Customer storage plan | +| `KAFKA_STORAGE_SIZE` | Volume size per Kafka broker | `100Gi` | [Size the clusters](#size-the-clusters) | +| `CLICKHOUSE_SHARDS` | ClickHouse shard count | `1` | ClickHouse profile, [Size the clusters](#size-the-clusters) | +| `CLICKHOUSE_REPLICAS` | ClickHouse replicas per shard | `3` | ClickHouse profile, [Size the clusters](#size-the-clusters) | +| `CLICKHOUSE_STORAGE_CLASS` | StorageClass for the ClickHouse and Keeper volumes | `` | Customer storage plan | +| `CLICKHOUSE_STORAGE_SIZE` | Data volume size per ClickHouse replica | `200Gi` | [Size the clusters](#size-the-clusters) | +| `CLICKHOUSE_SERVER_IMAGE` | Server image of the ClickHouse operator; Keeper runs from it | `/middleware/clickhouse-server:` | Printed by the command in [Logging](#logging) step 3, after the operator is installed | + +**Runtime secrets** — typed at a prompt when needed, never stored: + +| Variable | Meaning | Example | Where to get it | +| --- | --- | --- | --- | +| `AC_TOKEN` | Customer Portal access token for `violet ac` | (never stored) | Customer Portal, **Settings**; valid for 24 hours | +| `ACP_TOKEN` | Platform API token for `violet push` | (never stored) | Web console of the target platform, **Profile** > **API Tokens**; the account needs the `platform-admin-system` role | +| `REGISTRY_USERNAME` | Platform registry user of a `global` side: chosen in the installation request of that side (provider articles), used again to push packages to the standby | (never stored) | Decided by the delivery; record it in the customer's credential store, not in the delivery directory | +| `REGISTRY_PASSWORD` | Platform registry password of that `global` side, likewise | (never stored) | As above | +| `CH_ADMIN_SHA256` | Digest of the generated ClickHouse `admin` password, filled in when `73-clickhouse.yaml` is applied | (never stored) | Computed from the Secret `cpaas-clickhouse-passwords` by the apply command in [Logging](#logging) | +| `CH_LOGGING_SHA256` | Digest of the generated `platform-logging` password, likewise | (never stored) | As above | +| `SECRET_VALUE` | Any credential typed for a Secret in the [credential pattern](#credentials) | (never stored) | The system that owns the credential | + +#### Example files + +`site.env`: + +```bash +ACP_VERSION='v4.4.0' +PLATFORM_HOST='acp.example.com' +REGISTRY_DOMAIN='acp.example.com:11443' +K8S_VERSION='v1.35.' +TIMEZONE='Asia/Singapore' +NTP_SERVERS='192.0.2.123 192.0.2.124' +BOOT_SSH_PUBLIC_KEY='ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIExampleKeyOnly delivery@example.com' +MACHINE_CONFIG_VERSION='' +METALLB_MODULE_NAME='' +METALLB_VERSION='' +VICTORIAMETRICS_VERSION='' +LOGAGENT_VERSION='' +LOG_API_VERSION='' +INGRESS_NGINX_CHANNEL='stable' +INGRESS_NGINX_CSV='' +CLICKHOUSE_OPERATOR_CSV='' +# CHRONY_CONF_B64 is appended by the command in "Set NTP" +``` + +`wl-prod.env`: + +```bash +CLUSTER_NAME='wl-prod' +CONTROL_PLANE_VIP='192.0.2.20' +CLUSTER_CIDR='198.18.0.0/16' +SERVICE_CIDR='198.19.0.0/16' +KUBE_OVN_JOIN_CIDR='100.65.0.0/16' +INGRESS_VIP_RANGE='192.0.2.50/32' +INGRESS_VIP='192.0.2.50' +GLOBAL_KUBECONFIG='/root/.kube/acp/global-primary.kubeconfig' +CLUSTER_KUBECONFIG='/root/.kube/acp/wl-prod.kubeconfig' +MONITOR_SIZE='Medium' +MONITOR_STORAGE_GI='100' +MONITOR_RETENTION_DAYS='7' +MONITOR_STORAGE_NODES='["wl-prod-infra-1", "wl-prod-infra-2", "wl-prod-infra-3"]' +KAFKA_STORAGE_CLASS='' +KAFKA_STORAGE_SIZE='100Gi' +CLICKHOUSE_SHARDS='1' +CLICKHOUSE_REPLICAS='3' +CLICKHOUSE_STORAGE_CLASS='' +CLICKHOUSE_STORAGE_SIZE='200Gi' +# CLICKHOUSE_SERVER_IMAGE is appended after the ClickHouse operator is installed +# provider variables from the provider article follow +``` + +`global-standby.env` is a copy of `global-primary.env` in which `CLUSTER_NAME` +stays `global` and the addresses, CIDRs, node names and both kubeconfigs are the +standby's. + +### Delivery flow + +#### Multi-cluster with disaster recovery + +| Stage | Action | Pass criteria | +| ---: | --- | --- | +| 1 | Freeze the machine inventory | Counts and specs derived as in [Size the clusters](#size-the-clusters) and signed off | +| 2 | Freeze networks, VIPs, DNS, certificates and port approvals | No Pod, Service or join CIDR overlaps another CIDR of any cluster or any node, VIP or customer network; every VIP is an IP address; port approvals obtained per [Network flows and ports](#network-flows-and-ports) and tested | +| 3 | Fill `site.env` and every cluster file | Every variable filled with real values (`CLICKHOUSE_SERVER_IMAGE` follows in stage 8 or 14); rendering leaves no `${NAME}` placeholder | +| 4 | Prepare the provider: facts, credentials, provider packages — see your provider article | Provider prerequisites checked off | +| 5 | [Download the packages](#download-the-packages) with `violet` | `violet verify` passes for every package | +| 6 | Provision **two** bootstrap nodes, one per side; never reuse one | Each meets the [bootstrap node requirements](https://docs.alauda.io/immutable-infra/1.0/global/install.html#bootstrap-host-requirements): at least 4 cores, 8 GB, 300 GB disk (x86_64), static IP | +| 7 | Point the platform domain at the primary VIP; install the primary `global` — see your provider article | Installer reports success; all nodes `Ready`; `ClusterModule/global` healthy | +| 8 | On the primary: upload the packages, [enable label sync](#enable-label-synchronization), install the Machine Configuration plugin and [configure machines](#configure-machines), then monitoring and logging | Pools `UPDATED=True`; plugins `Running` on `infra` | +| 9 | Switch the platform domain to the standby VIP; install the standby `global` — see your provider article | As stage 7, on the standby | +| 10 | Repeat stage 8 on the standby with `global-standby.env` | As stage 8, on the standby | +| 11 | Switch the platform domain back to the primary VIP; install `etcd-sync` on the standby — see your provider article | Zero missed and zero surplus keys; the standby's plugins still `Running` | +| 12 | Decommission both bootstrap clusters after the provider handoff checks | Kubeconfigs saved; provider Secrets present on both sides | +| 13 | Create the workload clusters from `global`, including the customer's worker pools — see your provider article | Every pool matches the plan; labels and taints present on the nodes; at least one untainted worker pool | +| 14 | On each workload cluster: upload the operators, install the Machine Configuration plugin and configure machines, then MetalLB and Ingress NGINX, monitoring, logging | Each plugin verified on its intended pool | +| 15 | [Accept the delivery](#accept-the-delivery), run the failover drill, archive | Checklist passes | + +Stages 7 → 9 → 11 must run in that order, and what switches is the **platform +domain** — each cluster is always reached through its own VIP. Several platform +resources are rendered with the platform domain, so it must resolve to the side +being installed; it is switched back before `etcd-sync` because the primary is +the active source of the first synchronization and `etcd-sync` refuses to start +when the domain points at a cluster that runs it +([Install etcd-sync](https://docs.alauda.io/immutable-infra/1.0/global/install.html#install-etcd-sync)). + +Each side's `global` plugins are installed while the platform domain points at +that side (stages 8 and 10), before synchronization starts; each side's +monitoring and logging store only that side's data. Keep the machine +configuration identical on both sides (same `site.env`), so that both sides +converge to the same node configuration whichever side is active. + +#### Single-cluster variant + +One bootstrap node; skip stages 9 to 11 and 13. `global` also receives MetalLB and +Ingress NGINX on its `infra` pool (stage 8), with `INGRESS_VIP_RANGE` and +`INGRESS_VIP` set in `global-primary.env`, and the customer's worker pools for the +applications. + +#### Stop rules + +Stop and find the cause instead of deleting and re-creating resources: + +| Symptom | Stop | Look at first | +| --- | --- | --- | +| Machines stay `Provisioning` | Creating further pools | Provider conditions on the `Machine`, IaaS events, provider controller logs | +| `kubeadm` init or join fails | Further pools | Registry reachability, DNS, NTP, the endpoint VIP | +| API VIP unreachable | Plugin installation | VIP mode prerequisites, port approvals | +| Node without `providerID` or with a wrong IP | Acceptance | Provider resources (see the provider article) | +| Plugin Pod `Pending` or on the wrong pool | Acceptance | `nodeSelector`, taints, tolerations, label sync, local volume node affinity | +| etcd-sync reports differences | Any failover or uninstall of `etcd-sync` | [Failover](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#failover-procedure) prerequisites | + +Never delete the `Cluster`, `KubeadmControlPlane` or provider infrastructure +objects of a running cluster as a cleanup step: they own the live machines, and +deleting them deletes the nodes. + +### Install the platform plugins + +Install plugins only after the target pool's nodes are `Ready` and carry their +labels. Placement is set through each plugin's own parameters — never by patching +the Deployments or StatefulSets that a plugin or operator generates. Ingress NGINX +and MetalLB tolerate no taints by default, so their +`nodeSelector` and tolerations are always set explicitly. + +The manifests below are written for the consolidated `infra` layout. For the +dedicated layout, apply the diff shown after each manifest: replace `infra` with +`ingress`, `monitor` or `log` in both the selector and the toleration. + +`ModuleInfo` objects are created on `global` even when the target is a workload +cluster. The platform renames a `ModuleInfo` after creation, so apply each one +once; later changes go through the console or the renamed object. + +The commands show the workload cluster `wl-prod`. For any other cluster, load its +file and use its directory — for example `load_env global-primary.env` and +`global-primary/65-victoriametrics.yaml` for the primary `global`. + +#### Enable label synchronization + +Once per `global` side, before the first plugin, make `capi-controller-manager` +sync the role labels from `Machine` to `Node`: + +```bash +# on global (each side) +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get deploy capi-controller-manager \ + -o jsonpath='{.spec.template.spec.containers[0].args}'; echo +# only if no --additional-sync-machine-labels argument is present yet: +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system patch deploy capi-controller-manager --type json \ + -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--additional-sync-machine-labels=node-role.kubernetes.io/.*"}]' +``` + +If the argument already exists with other labels, extend its value instead of +adding a second argument. Verify on the `Node`, not only on the `Machine`: + +```bash +# on the cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes -l node-role.kubernetes.io/infra +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes \ + -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key' +``` + +#### Download the packages + +On a machine with Customer Portal access, install **Violet v4.2.13 or later** +([Download Packages](https://docs.alauda.io/container_platform/4.4/extend/download_package.html)) +and download the Core Package and the **Common ACP Extensions** scenario for the +same version and architecture: + +```bash +# on the download machine +read -rsp 'Customer Portal access token: ' AC_TOKEN; echo +violet ac login --access-token="${AC_TOKEN}" +unset AC_TOKEN +violet ac download-pkg --arch=x86 --platformVersion="${ACP_VERSION}" +violet ac scenarios --arch=amd64 --platformVersion="${ACP_VERSION}" +violet ac download-app --arch=amd64 --platformVersion="${ACP_VERSION}" --scenario="Common ACP Extensions" +violet ac packages --arch=amd64 --platformVersion="${ACP_VERSION}" --scenario="Common ACP Extensions" +``` + +Copy only the eight required Aligned extensions into the Core Package's +`plugins/` directory, as listed in +[Required Aligned Extensions](https://docs.alauda.io/container_platform/4.4/install/prepare/download.html#required-aligned-extensions); +everything else is uploaded after installation. Make sure the following packages +are present, and download any that the scenario does not contain by application +ID (`violet ac download-app --appID= --appVersion=`) or from the +Customer Portal: + +| Package | Kind | Used for | +| --- | --- | --- | +| Alauda Container Platform Machine Configuration | Cluster plugin; released independently of ACP | Every cluster | +| MetalLB | Cluster plugin | Clusters with application ingress | +| Alauda Build of Ingress NGINX Controller | Operator | Clusters with application ingress | +| ACP Monitoring with VictoriaMetrics | Cluster plugin | Every cluster | +| Alauda Streaming Service for Kafka and Alauda Container Platform Data Services RDS Framework | Operators | Every cluster (log storage) | +| Alauda Data Services Analytical Database E1 (ClickHouse, package `clickhouse-operator`) | Operator | Every cluster (log storage) | +| Alauda Container Platform Log Storage Management | Installed from **Cluster Plugins** ([logging installation](https://docs.alauda.io/logging-service/4.4/install/index.html)) | Every cluster | +| Alauda Container Platform Log Collector | Cluster plugin | Every cluster | +| Alauda Container Platform Log Essentials | Cluster plugin | `global` only | +| Alauda Container Platform etcd Synchronizer (`etcd-sync`) | Cluster plugin | Standby `global` (provider article) | + +Verify the signatures, then move the packages to the delivery host: + +```bash +# on the download machine +violet verify +``` + +#### Upload the packages + +Keep operator packages in `packages/operators/` and cluster plugins in +`packages/plugins/`. A cluster plugin is uploaded once per platform and +distributed by its cluster affinity; an operator is uploaded to every cluster +that runs it ([Upload Packages](https://docs.alauda.io/container_platform/4.4/extend/upload_package.html#violet_push_usage)). + +```bash +# on the delivery host — primary global, while the platform domain points at it +load_env global-primary.env +read -rsp 'Platform API token (primary): ' ACP_TOKEN; echo +violet push packages/plugins --platform-address "https://${PLATFORM_HOST}" --platform-token "${ACP_TOKEN}" +violet push packages/operators --platform-address "https://${PLATFORM_HOST}" --platform-token "${ACP_TOKEN}" +unset ACP_TOKEN +``` + +The standby `global` has its own registry, so it receives every package as well. +Address it through its own VIP and registry, as the upload page requires for a +standby; otherwise the images land in the primary's registry: + +```bash +# on the delivery host — standby global +load_env global-standby.env +read -rsp 'Platform API token (standby): ' ACP_TOKEN; echo +read -rp 'Standby registry user: ' REGISTRY_USERNAME +read -rsp 'Standby registry password: ' REGISTRY_PASSWORD; echo +for dir in packages/plugins packages/operators; do + violet push "$dir" --platform-address "https://${CONTROL_PLANE_VIP}" --platform-token "${ACP_TOKEN}" \ + --dest-repo "${CONTROL_PLANE_VIP}:11443" --username "${REGISTRY_USERNAME}" --password "${REGISTRY_PASSWORD}" +done +unset ACP_TOKEN REGISTRY_USERNAME REGISTRY_PASSWORD +``` + +After each workload cluster is created, upload the operators to it: + +```bash +# on the delivery host — one workload cluster +load_env wl-prod.env +read -rsp 'Platform API token: ' ACP_TOKEN; echo +violet push packages/operators --platform-address "https://${PLATFORM_HOST}" --platform-token "${ACP_TOKEN}" \ + --clusters "${CLUSTER_NAME}" +unset ACP_TOKEN +``` + +`violet` documents no environment-variable or file input for the token and the +registry password, so they are passed as flags from shell variables: they stay out +of shell history but are visible in the process list while `violet` runs, so use +a delivery host that only the delivery team can log in to. The platform needs a +few minutes to synchronize an uploaded version before it can be installed. + +#### Look up plugin versions + +Read the published names and versions on `global` once and record them in +`site.env`. Install the same version on every cluster of the delivery. + +```bash +# on global +for m in machine-config victoriametrics logagent log-api; do + printf '%s: ' "$m" + kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get moduleconfigs -l cpaas.io/module-name="$m" \ + -o jsonpath='{range .items[*]}{.spec.version}{" "}{end}'; echo +done +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get moduleplugins | grep -i metallb # -> METALLB_MODULE_NAME +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get moduleconfigs -l cpaas.io/module-name="${METALLB_MODULE_NAME}" \ + -o jsonpath='{range .items[*]}{.spec.version}{"\t"}{.spec.config}{"\n"}{end}' +``` + +For the operators installed by `Subscription` (Ingress NGINX and ClickHouse), read +the catalog source, channel and CSV on a cluster they were uploaded to: + +```bash +# on the workload cluster +for p in ingress-nginx-operator clickhouse-operator; do + kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get packagemanifest "$p" -o json | \ + jq -r '.status.catalogSource as $src | .status.channels[] | .name as $ch | .entries[] | [$src, $ch, .name, .version] | @tsv' +done +``` + +If the catalog source is not `platform`, set `source` in `63` and `70` to it. + +#### Machine Configuration + +Install it first, on every cluster: until it is installed, the `MachineConfig`, +`MachineConfigPool` and `MachineConfiguration` APIs do not exist +([Installing Machine Configuration](https://docs.alauda.io/immutable-infra/1.0/machine-configuration/install.html)). +The plugin takes no configuration parameters. + +`/50-machine-config.yaml` — apply on `global`: + +```yaml +apiVersion: cluster.alauda.io/v1alpha1 +kind: ModuleInfo +metadata: + labels: + cpaas.io/cluster-name: ${CLUSTER_NAME} + cpaas.io/module-name: machine-config + cpaas.io/module-type: plugin + name: ${CLUSTER_NAME}-machine-config +spec: + config: {} + version: ${MACHINE_CONFIG_VERSION} +``` + +```bash +# on global +apply_rendered wl-prod.env wl-prod/50-machine-config.yaml "${GLOBAL_KUBECONFIG}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get moduleinfo -l cpaas.io/module-name=machine-config # STATUS Running +# on the cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get deployment machine-config-controller +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get daemonset machine-config-daemon +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get machineconfigpool # master and worker, MACHINECOUNT = node counts +``` + +The `worker` pool's machine count must include every `infra` (or dedicated) node; +a node missing from it lacks the `node-role.kubernetes.io/worker` label. Continue +with [Configure machines](#configure-machines) now, before the other plugins, so +that every later component starts with the agreed time zone and time source. + +#### MetalLB + +On clusters with application ingress: the workload clusters, and `global` in the +single-cluster variant. Confirm that the MetalLB `ModuleConfig` has an empty +`spec.config` (see [Look up plugin versions](#look-up-plugin-versions)); if it +has a form, install from **Administrator** > **Marketplace** > **Cluster +Plugins** instead +([Installing the MetalLB Operator](https://docs.alauda.io/container_platform/4.4/networking/operators/metallb_operator.html#installing_and_uninstalling_metallb)). + +`/60-metallb.yaml` — apply on `global`: + +```yaml +apiVersion: cluster.alauda.io/v1alpha1 +kind: ModuleInfo +metadata: + labels: + cpaas.io/cluster-name: ${CLUSTER_NAME} + cpaas.io/module-name: ${METALLB_MODULE_NAME} + cpaas.io/module-type: plugin + name: ${CLUSTER_NAME}-metallb +spec: + config: {} + version: ${METALLB_VERSION} +``` + +Placement is configured on the `MetalLB` resource that the plugin creates. Its +four fields cover both components: `controllerNodeSelector` and +`controllerTolerations` the controller, `nodeSelector` and `speakerTolerations` +the speaker. Without the speaker toleration the speaker cannot run on the pool and +the VIP is never announced +([Configure Component Placement](https://docs.alauda.io/container_platform/4.4/networking/operators/metallb_operator.html#configure-component-placement)). + +`/61-metallb-placement.yaml` — apply on the cluster: + +```yaml +apiVersion: metallb.io/v1beta1 +kind: MetalLB +metadata: + name: metallb + namespace: metallb-system +spec: + controllerNodeSelector: + node-role.kubernetes.io/infra: "" + controllerTolerations: + - key: node-role.kubernetes.io/infra + operator: Exists + effect: NoSchedule + nodeSelector: + node-role.kubernetes.io/infra: "" + speakerTolerations: + - key: node-role.kubernetes.io/infra + operator: Exists + effect: NoSchedule +``` + +The address pool is reserved for the ingress VIP (`autoAssign: false`, so no other +`LoadBalancer` Service can take it), and the Layer-2 advertisement is limited to +the pool that runs the ingress controller, so that the node announcing the VIP is +one that can serve it +([Configure MetalLB](https://docs.alauda.io/container_platform/4.4/networking/functions/configure_metallb.html)). + +`/62-metallb-ingress-pool.yaml` — apply on the cluster: + +```yaml +apiVersion: metallb.io/v1beta1 +kind: IPAddressPool +metadata: + name: ingress-vip + namespace: metallb-system +spec: + addresses: + - ${INGRESS_VIP_RANGE} + autoAssign: false + avoidBuggyIPs: true +--- +apiVersion: metallb.io/v1beta1 +kind: L2Advertisement +metadata: + name: ingress-vip + namespace: metallb-system +spec: + ipAddressPools: + - ingress-vip + nodeSelectors: + - matchLabels: + node-role.kubernetes.io/infra: "" +``` + +```bash +# on global +apply_rendered wl-prod.env wl-prod/60-metallb.yaml "${GLOBAL_KUBECONFIG}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get moduleinfo -l cpaas.io/module-name="${METALLB_MODULE_NAME}" +# on the workload cluster, once the MetalLB resource exists +apply_rendered wl-prod.env wl-prod/61-metallb-placement.yaml "${CLUSTER_KUBECONFIG}" +apply_rendered wl-prod.env wl-prod/62-metallb-ingress-pool.yaml "${CLUSTER_KUBECONFIG}" +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n metallb-system get pods -o wide # controller and speakers on infra nodes +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n metallb-system get ipaddresspool,l2advertisement +``` + +Dedicated pools — in `61`, and likewise for `nodeSelector`, `speakerTolerations` +and the `L2Advertisement` selector in `62`: + +```diff + spec: + controllerNodeSelector: +- node-role.kubernetes.io/infra: "" ++ node-role.kubernetes.io/ingress: "" + controllerTolerations: +- - key: node-role.kubernetes.io/infra ++ - key: node-role.kubernetes.io/ingress + operator: Exists + effect: NoSchedule +``` + +#### Ingress NGINX + +The operator is installed from OperatorHub through a `Subscription` on the target +cluster; Alauda operators support only `Manual` approval +([Operator](https://docs.alauda.io/container_platform/4.4/extend/operator.html)). +`spec.config` places the operator Pod itself on the pool. + +`/63-ingress-nginx-operator.yaml` — apply on the cluster: + +```yaml +apiVersion: operators.coreos.com/v1alpha1 +kind: Subscription +metadata: + annotations: + cpaas.io/target-namespaces: "" + name: ingress-nginx-operator + namespace: ingress-nginx-operator +spec: + channel: ${INGRESS_NGINX_CHANNEL} + installPlanApproval: Manual + name: ingress-nginx-operator + source: platform + sourceNamespace: cpaas-system + startingCSV: ${INGRESS_NGINX_CSV} + config: + nodeSelector: + node-role.kubernetes.io/infra: "" + tolerations: + - key: node-role.kubernetes.io/infra + operator: Exists + effect: NoSchedule +``` + +```bash +# on the workload cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" create namespace ingress-nginx-operator +apply_rendered wl-prod.env wl-prod/63-ingress-nginx-operator.yaml "${CLUSTER_KUBECONFIG}" +PLAN=$(kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n ingress-nginx-operator get subscription ingress-nginx-operator \ + -o jsonpath='{.status.installPlanRef.name}') +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n ingress-nginx-operator patch installplan "$PLAN" --type=json \ + -p='[{"op": "replace", "path": "/spec/approved", "value": true}]' +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n ingress-nginx-operator get csv # PHASE Succeeded +``` + +The instance uses a `LoadBalancer` Service that requests the ingress VIP from the +MetalLB pool. `externalTrafficPolicy: Local` preserves the client source IP; it +requires the controller's `nodeSelector` to match the `L2Advertisement` node +selector, which `62` already does +([Tasks for Ingress NGINX](https://docs.alauda.io/container_platform/4.4/networking/how_to/tasks_for_ingress_nginx.html)). +The operator creates the IngressClass `ingress-nginx-operator-apps` for this +instance (namespace and name of the resource). + +`/64-ingress-nginx.yaml` — apply on the cluster: + +```yaml +apiVersion: ingress-nginx.alauda.io/v1 +kind: IngressNginx +metadata: + name: apps + namespace: ingress-nginx-operator +spec: + controller: + hostNetwork: false + replicaCount: 2 + nodeSelector: + node-role.kubernetes.io/infra: "" + tolerations: + - key: node-role.kubernetes.io/infra + operator: Exists + effect: NoSchedule + service: + type: LoadBalancer + externalTrafficPolicy: Local + annotations: + metallb.universe.tf/address-pool: ingress-vip + metallb.universe.tf/loadBalancerIPs: "${INGRESS_VIP}" +``` + +```bash +# on the workload cluster +apply_rendered wl-prod.env wl-prod/64-ingress-nginx.yaml "${CLUSTER_KUBECONFIG}" +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n ingress-nginx-operator get pods -o wide # controllers on infra nodes +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n ingress-nginx-operator get svc # EXTERNAL-IP = INGRESS_VIP +curl -s -o /dev/null -w '%{http_code}\n' "http://${INGRESS_VIP}/" # 404 from the default backend +``` + +Dedicated pools: the same `infra` → `ingress` change as for MetalLB, in `63` and +`64`. + +With a customer's commercial load balancer instead of MetalLB, skip `60` to `62`, +remove `service` from `64`, and run the controller on the host network so that +the load balancer forwards to ports 80 and 443 of the pool's nodes: + +```diff + controller: +- hostNetwork: false ++ hostNetwork: true +- service: +- type: LoadBalancer +- externalTrafficPolicy: Local +- annotations: +- metallb.universe.tf/address-pool: ingress-vip +- metallb.universe.tf/loadBalancerIPs: "${INGRESS_VIP}" +``` + +If the node firewall does not admit those ports, open them with +[Managing Firewall Ports](https://docs.alauda.io/immutable-infra/1.0/machine-configuration/firewall.html) +rather than by changing nodes by hand. + +#### Monitoring + +Every cluster installs the full VictoriaMetrics stack (`agentOnly: false`) and +keeps its data on `LocalVolume` storage under the persistent `/var/cpaas` disk — +the default `/cpaas/monitoring` is not writable on Alauda OS +([LocalVolume Storage Path on Immutable Infrastructure](https://docs.alauda.io/container_platform/4.4/observability/monitor/install_monitor.html#immutable_storage_path)). +Every node in `MONITOR_STORAGE_NODES` must belong to the monitoring pool and have +the persistent disk; the number of nodes is the number of vmstorage replicas. For a +workload cluster, `global` must reach port 11780 of the workload cluster before +you install. + +`/65-victoriametrics.yaml` — apply on `global`: + +```yaml +kind: ModuleInfo +apiVersion: cluster.alauda.io/v1alpha1 +metadata: + name: ${CLUSTER_NAME}-victoriametrics + labels: + cpaas.io/cluster-name: ${CLUSTER_NAME} + cpaas.io/module-name: victoriametrics + cpaas.io/module-type: plugin +spec: + version: ${VICTORIAMETRICS_VERSION} + config: + storage: + type: LocalVolume + capacity: ${MONITOR_STORAGE_GI} + nodes: ${MONITOR_STORAGE_NODES} + path: /var/cpaas/monitoring + storageClass: "" + pvSelectorK: "" + pvSelectorV: "" + replicas: 1 + agentOnly: false + agentReplicas: 1 + crossClusterDependency: + victoriametrics: "" + components: + nodeSelector: + - key: node-role.kubernetes.io/infra + value: "" + tolerations: + - effect: NoSchedule + key: node-role.kubernetes.io/infra + operator: Exists + nodeExporter: + port: 9100 + resources: null + vmstorage: + retention: ${MONITOR_RETENTION_DAYS} + resources: null + kubeStateExporter: + resources: null + vmalert: + resources: null + prometheusAdapter: + resources: null + vmagent: + scrapeInterval: 60 + scrapeTimeout: 45 + resources: null + vminsert: + resources: null + alertmanager: + resources: null + vmselect: + resources: null + size: ${MONITOR_SIZE} +``` + +`components.nodeSelector` is a list of `key`/`value` entries, not a map. + +```bash +# on global +apply_rendered wl-prod.env wl-prod/65-victoriametrics.yaml "${GLOBAL_KUBECONFIG}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get moduleinfo -l cpaas.io/module-name=victoriametrics # STATUS Running +# on the cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get pods -o wide | \ + grep -E 'vmstorage|vminsert|vmselect|vmagent|vmalert|alertmanager' # on infra nodes +``` + +Dedicated pools: + +```diff + nodeSelector: +- - key: node-role.kubernetes.io/infra ++ - key: node-role.kubernetes.io/monitor + value: "" + tolerations: + - effect: NoSchedule +- key: node-role.kubernetes.io/infra ++ key: node-role.kubernetes.io/monitor + operator: Exists +``` + +#### Logging + +Every cluster is its own log storage cluster: it runs Kafka, ClickHouse and the +Log Storage Management operator, and its collector writes to itself. Log +Essentials is installed once on `global` (on each side of the DR pair). Follow the +order below; the product reference is the logging service +[Installation](https://docs.alauda.io/logging-service/4.4/install/index.html). + +**1. Operators.** Kafka and ClickHouse are OLM operators, uploaded with the other +operators; their operator Pods run on the logging pool. + +The ClickHouse operator is the package `clickhouse-operator` on channel `stable`, +installed into the `operators` namespace, where the Keeper step below reads its +server image +([Installation](https://docs.alauda.io/clickhouse/4.4/installation.html)). + +`/70-clickhouse-operator.yaml` — apply on the cluster: + +```yaml +apiVersion: operators.coreos.com/v1alpha1 +kind: Subscription +metadata: + annotations: + cpaas.io/target-namespaces: "" + name: clickhouse-operator + namespace: operators +spec: + channel: stable + installPlanApproval: Manual + name: clickhouse-operator + source: platform + sourceNamespace: cpaas-system + startingCSV: ${CLICKHOUSE_OPERATOR_CSV} + config: + nodeSelector: + node-role.kubernetes.io/infra: "" + tolerations: + - key: node-role.kubernetes.io/infra + operator: Exists + effect: NoSchedule +``` + +```bash +# on the cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get namespace operators || \ + kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" create namespace operators +apply_rendered wl-prod.env wl-prod/70-clickhouse-operator.yaml "${CLUSTER_KUBECONFIG}" +PLAN=$(kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n operators get subscription clickhouse-operator \ + -o jsonpath='{.status.installPlanRef.name}') +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n operators patch installplan "$PLAN" --type=json \ + -p='[{"op": "replace", "path": "/spec/approved", "value": true}]' +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n operators get deploy clickhouse-operator -o wide # READY 1/1, on an infra node +``` + +The Kafka documentation describes only a console installation and does not name +the packages, so the two Kafka operators remain a console step +([Install](https://docs.alauda.io/kafka/4.3/installation.html)). On the cluster, +in **Marketplace** > **OperatorHub**, install **Alauda Container Platform Data +Services RDS Framework** first — Kafka 4.2.0 requires version 4.2.2 or later +([Upgrade](https://docs.alauda.io/kafka/4.3/upgrade.html)) — and then **Alauda +Streaming Service for Kafka**, each with upgrade strategy **Manual**, installation +mode **Cluster**, channel **stable** and the recommended namespace. Then move the +Kafka operator Pod to the logging pool through its `Subscription`, as the +[Operator](https://docs.alauda.io/container_platform/4.4/extend/operator.html) +page describes for an installed operator: + +```bash +# on the cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get subscriptions -A | grep -i kafka # note NAMESPACE and NAME +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n patch subscription --type merge -p \ + '{"spec":{"config":{"nodeSelector":{"node-role.kubernetes.io/infra":""},"tolerations":[{"key":"node-role.kubernetes.io/infra","operator":"Exists","effect":"NoSchedule"}]}}}' +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get crd rdskafkas.middleware.alauda.io +``` + +**2. Kafka.** Create the SASL password Secret directly from a generated value — +it is never displayed or stored elsewhere. On Alauda OS the password must be at +least 32 characters: + +```bash +# on the cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system create secret generic platform-logging-password \ + --from-file=password=<(openssl rand -hex 16 | tr -d '\n') +``` + +The Kafka manifest uses the product's production values (three brokers, a +three-member KRaft controller quorum, replication factor 3). The product manifest +tolerates every taint so that evaluation clusters can schedule; here the +toleration is narrowed to the logging pool. The five broker settings marked +mandatory are required for audit batches and must not be changed. + +`/71-kafka.yaml` — apply on the cluster: + +```yaml +apiVersion: middleware.alauda.io/v1 +kind: RdsKafka +metadata: + name: cpaas-kafka + namespace: cpaas-system +spec: + version: 4.2.0 + mode: KRaft + replicas: 3 + upgradeOption: + autoUpgrade: false + storage: + class: ${KAFKA_STORAGE_CLASS} + size: ${KAFKA_STORAGE_SIZE} + resources: + limits: + cpu: "2" + memory: 4Gi + requests: + cpu: 200m + memory: 1Gi + config: + message.max.bytes: "10485760" # mandatory + replica.fetch.max.bytes: "10485760" # mandatory + socket.request.max.bytes: "104857600" # mandatory + auto.create.topics.enable: "false" # mandatory + unclean.leader.election.enable: "false" # mandatory + default.replication.factor: "3" + min.insync.replicas: "2" + offsets.topic.replication.factor: "3" + transaction.state.log.replication.factor: "3" + transaction.state.log.min.isr: "2" + log.retention.ms: "172800000" + log.retention.bytes: "4928307200" + num.partitions: "30" + controller: + replicas: 3 + roles: [controller] + storage: + class: ${KAFKA_STORAGE_CLASS} + size: 5Gi + resources: + limits: + cpu: "1" + memory: 2Gi + requests: + cpu: 200m + memory: 512Mi + template: + pod: + tolerations: + - key: node-role.kubernetes.io/infra + operator: Exists + effect: NoSchedule + kafka: + authorization: + type: simple + listeners: + plain: + authentication: + type: scram-sha-512 + template: + pod: + tolerations: + - key: node-role.kubernetes.io/infra + operator: Exists + effect: NoSchedule + entityOperator: + topicOperator: + resources: + limits: + cpu: 500m + memory: 1Gi + requests: + cpu: 100m + memory: 256Mi + userOperator: + resources: + limits: + cpu: 500m + memory: 1Gi + requests: + cpu: 100m + memory: 256Mi + template: + pod: + tolerations: + - key: node-role.kubernetes.io/infra + operator: Exists + effect: NoSchedule + kafkaExporter: + resources: + limits: + cpu: 200m + memory: 256Mi + requests: + cpu: 100m + memory: 128Mi + template: + pod: + tolerations: + - key: node-role.kubernetes.io/infra + operator: Exists + effect: NoSchedule +--- +apiVersion: middleware.alauda.io/v1 +kind: RdsKafkaUser +metadata: + name: platform-logging + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + authentication: + type: scram-sha-512 + password: + # read from the Secret created above, never a literal value + valueFrom: + secretKeyRef: + name: platform-logging-password + key: password + authorization: + type: simple + acls: + - resource: {type: topic, name: ALAUDA_LOG_TOPIC, patternType: literal} + operation: All + host: "*" + - resource: {type: topic, name: ALAUDA_EVENT_TOPIC, patternType: literal} + operation: All + host: "*" + - resource: {type: topic, name: ALAUDA_AUDIT_TOPIC, patternType: literal} + operation: All + host: "*" + - resource: {type: group, name: alauda_log, patternType: literal} + operation: Read + host: "*" + - resource: {type: group, name: alauda_event, patternType: literal} + operation: Read + host: "*" + - resource: {type: group, name: alauda_audit, patternType: literal} + operation: Read + host: "*" + - resource: {type: group, name: razor-, patternType: prefix} + operation: Read + host: "*" + - resource: {type: group, name: logforward-, patternType: prefix} + operation: Read + host: "*" + - resource: {type: cluster, name: kafka-cluster, patternType: literal} + operation: DescribeConfigs + host: "*" +--- +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-log-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_LOG_TOPIC + partitions: 30 + replicas: 3 + config: + retention.ms: "172800000" + max.message.bytes: "10485760" +--- +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-event-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_EVENT_TOPIC + partitions: 30 + replicas: 3 + config: + retention.ms: "172800000" + max.message.bytes: "10485760" +--- +apiVersion: middleware.alauda.io/v1 +kind: RdsTopic +metadata: + name: alauda-audit-topic + namespace: cpaas-system + labels: + middleware.alauda.io/cluster: cpaas-kafka +spec: + topicName: ALAUDA_AUDIT_TOPIC + partitions: 30 + replicas: 3 + config: + retention.ms: "172800000" + max.message.bytes: "10485760" +``` + +The product manifest exposes only tolerations for Kafka placement: they admit +Kafka to the logging pool but do not confine it there. Check where the brokers +land and that their volumes follow the storage plan. + +```bash +# on the cluster +apply_rendered wl-prod.env wl-prod/71-kafka.yaml "${CLUSTER_KUBECONFIG}" +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get rdskafka cpaas-kafka -o jsonpath='{.status.phase}{"\n"}' # Active +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get pods -o wide | grep cpaas-kafka +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get svc cpaas-kafka-kafka-bootstrap +``` + +**3. ClickHouse.** The ClickHouse documentation builds the cluster the logging +stack expects: a three-member Keeper quorum for replication, and a +`ClickHouseInstallation` with the fixed names instance `cpaas-clickhouse` +(Service `clickhouse-cpaas-clickhouse`), cluster `replicated`, database +`observability` and account `platform-logging` +([Deploy as ACP logging storage](https://docs.alauda.io/clickhouse/4.4/how_to/deploy_as_logging_storage.html)). +Do not rename them: the connection Secret in step 5 refers to them. Keeper and +ClickHouse keep their data on volumes of `CLICKHOUSE_STORAGE_CLASS`, one Keeper +member and one replica per logging node. + +Generate the two account passwords and the inter-replica secret into Secrets. +They are never displayed; later steps read them from there: + +```bash +# on the cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system create secret generic cpaas-clickhouse-passwords \ + --from-file=admin=<(openssl rand -hex 16 | tr -d '\n') --from-file=logging=<(openssl rand -hex 16 | tr -d '\n') +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system create secret generic cpaas-clickhouse-cluster-secret \ + --from-file=secret=<(openssl rand -hex 24 | tr -d '\n') +``` + +Keeper runs from the server image that the operator uses, so nothing extra is +mirrored. Record that image in the cluster file once the operator runs +([Deploy a Keeper quorum](https://docs.alauda.io/clickhouse/4.4/how_to/deploy_keeper_quorum.html)): + +```bash +# on the cluster +OP=$(kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n operators get deploy clickhouse-operator \ + -o jsonpath='{.spec.template.spec.containers[?(@.name=="clickhouse-operator")].image}') +TAG=$(kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n operators get deploy clickhouse-operator \ + -o jsonpath='{.spec.template.spec.containers[?(@.name=="clickhouse-operator")].env[?(@.name=="CK_SERVER_IMAGE")].value}') +# on the delivery host: record it in the cluster file +sed -i '/^CLICKHOUSE_SERVER_IMAGE=/d' wl-prod.env +printf "CLICKHOUSE_SERVER_IMAGE='%s'\n" "${OP%%/*}/$TAG" >> wl-prod.env +``` + +`/72-keeper.yaml` — apply on the cluster. The `${HOSTNAME##*-}` in the +container command is a shell expansion that `render` leaves alone; each member +derives its `server_id` from its Pod ordinal: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: keeper + namespace: cpaas-system + labels: + app: keeper +data: + keeper_config.xml: | + + information1 + 0.0.0.0 + 256 + + 2181 + __SERVER_ID__ + /var/lib/clickhouse-keeper/coordination/log + /var/lib/clickhouse-keeper/coordination/snapshots + + 10000 + 30000 + information + + + 1keeper-0.keeper-headless9234 + 2keeper-1.keeper-headless9234 + 3keeper-2.keeper-headless9234 + + + +--- +apiVersion: v1 +kind: Service +metadata: + name: keeper-headless + namespace: cpaas-system + labels: + app: keeper +spec: + clusterIP: None + publishNotReadyAddresses: true + selector: + app: keeper + ports: + - name: client + port: 2181 + - name: raft + port: 9234 +--- +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: keeper + namespace: cpaas-system +spec: + maxUnavailable: 1 + selector: + matchLabels: + app: keeper +--- +apiVersion: apps/v1 +kind: StatefulSet +metadata: + name: keeper + namespace: cpaas-system + labels: + app: keeper +spec: + serviceName: keeper-headless + replicas: 3 + podManagementPolicy: Parallel + selector: + matchLabels: + app: keeper + template: + metadata: + labels: + app: keeper + spec: + tolerations: + - key: node-role.kubernetes.io/infra + operator: Exists + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra + operator: Exists + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - topologyKey: kubernetes.io/hostname + labelSelector: + matchLabels: + app: keeper + securityContext: + fsGroup: 101 + containers: + - name: keeper + image: ${CLICKHOUSE_SERVER_IMAGE} + imagePullPolicy: IfNotPresent + command: + - /bin/sh + - -c + - >- + ORD=${HOSTNAME##*-}; + sed "s/__SERVER_ID__/$((ORD+1))/" /etc/keeper-tpl/keeper_config.xml + > /var/lib/clickhouse-keeper/keeper_config.xml; + exec clickhouse-keeper --config-file=/var/lib/clickhouse-keeper/keeper_config.xml + ports: + - name: client + containerPort: 2181 + - name: raft + containerPort: 9234 + securityContext: + runAsNonRoot: true + runAsUser: 101 + runAsGroup: 101 + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + seccompProfile: + type: RuntimeDefault + resources: + requests: + cpu: 500m + memory: 512Mi + limits: + cpu: "2" + memory: 2Gi + readinessProbe: + tcpSocket: + port: 2181 + initialDelaySeconds: 5 + periodSeconds: 5 + failureThreshold: 30 + volumeMounts: + - name: config + mountPath: /etc/keeper-tpl + - name: data + mountPath: /var/lib/clickhouse-keeper + volumes: + - name: config + configMap: + name: keeper + volumeClaimTemplates: + - metadata: + name: data + spec: + accessModes: + - ReadWriteOnce + storageClassName: ${CLICKHOUSE_STORAGE_CLASS} + resources: + requests: + storage: 10Gi +``` + +A member that has lost its quorum still reports `Ready`, so check the quorum +itself: exactly one member is `leader` with `zk_synced_followers 2`, the other two +are `follower`. + +```bash +# on the cluster +apply_rendered wl-prod.env wl-prod/72-keeper.yaml "${CLUSTER_KUBECONFIG}" +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system rollout status statefulset/keeper +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get pods -l app=keeper -o wide # three infra nodes +for i in 0 1 2; do + kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system exec keeper-$i -- bash -c \ + 'exec 3<>/dev/tcp/127.0.0.1/2181; printf mntr >&3; timeout 5 cat <&3' | grep -E 'zk_server_state|zk_synced_followers' +done +``` + +`/73-clickhouse.yaml` — apply on the cluster with the command below it. +The two `${...SHA256}` placeholders are intentional: they are filled at apply +time with the SHA-256 digests of the generated passwords, so neither a password +nor a digest is ever written to a file. + +```yaml +apiVersion: clickhouse.altinity.com/v1 +kind: ClickHouseInstallation +metadata: + name: cpaas-clickhouse + namespace: cpaas-system +spec: + defaults: + templates: + dataVolumeClaimTemplate: data-volume + podTemplate: ch-pod + serviceTemplate: internal + configuration: + zookeeper: + nodes: + - host: keeper-0.keeper-headless.cpaas-system + - host: keeper-1.keeper-headless.cpaas-system + - host: keeper-2.keeper-headless.cpaas-system + clusters: + - name: replicated + secret: + valueFrom: + secretKeyRef: + name: cpaas-clickhouse-cluster-secret + key: secret + layout: + shardsCount: ${CLICKHOUSE_SHARDS} + replicasCount: ${CLICKHOUSE_REPLICAS} + users: + admin/password_sha256_hex: ${CH_ADMIN_SHA256} + admin/networks/ip: + - "::/0" + admin/access_management: "1" + platform-logging/password_sha256_hex: ${CH_LOGGING_SHA256} + platform-logging/networks/ip: + - "::/0" + platform-logging/grants/query: + - "GRANT ALL ON observability.*" + - "GRANT CLUSTER ON *.*" + templates: + serviceTemplates: + - name: internal + spec: + type: ClusterIP + ports: + - name: http + port: 8123 + - name: tcp + port: 9000 + podTemplates: + - name: ch-pod + podDistribution: + - type: ShardAntiAffinity + spec: + tolerations: + - key: node-role.kubernetes.io/infra + operator: Exists + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node-role.kubernetes.io/infra + operator: Exists + securityContext: + runAsUser: 101 + runAsGroup: 101 + fsGroup: 101 + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault + containers: + - name: clickhouse + resources: + requests: + cpu: "2" + memory: 8Gi + limits: + cpu: "4" + memory: 16Gi + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL + volumeClaimTemplates: + - name: data-volume + spec: + accessModes: + - ReadWriteOnce + storageClassName: ${CLICKHOUSE_STORAGE_CLASS} + resources: + requests: + storage: ${CLICKHOUSE_STORAGE_SIZE} +``` + +The pieces the logging stack depends on: the cluster secret is referenced, not +`secret: auto`, because on ACP the operator cannot create Secrets and the Pods +would stay in `CreateContainerConfigError`; `networks/ip` lets other Pods +connect; the `ClusterIP` Service template replaces the default `LoadBalancer` +Service, which in-cluster clients do not need; `GRANT CLUSTER` is needed because +the logging schema migrations use `ON CLUSTER`. No image is set — the operator +supplies its own. + +```bash +# on the cluster +ch_sha256() { # SHA-256 of one generated password; nothing is printed + kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get secret cpaas-clickhouse-passwords \ + -o jsonpath="{.data.$1}" | base64 -d | sha256sum | cut -d' ' -f1 +} +render wl-prod.env wl-prod/73-clickhouse.yaml | \ + sed -e "s/\${CH_ADMIN_SHA256}/$(ch_sha256 admin)/" -e "s/\${CH_LOGGING_SHA256}/$(ch_sha256 logging)/" | \ + kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" apply -f - +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get statefulset -l clickhouse.altinity.com/chi=cpaas-clickhouse \ + -o custom-columns=NAME:.metadata.name,DESIRED:.spec.replicas,READY:.status.readyReplicas # one StatefulSet per replica, each READY 1 +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get chi cpaas-clickhouse # STATUS Completed +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get pods -l clickhouse.altinity.com/chi=cpaas-clickhouse -o wide # one replica per infra node +``` + +Then create the `observability` database, which must exist before the storage +operator runs, and prove that `platform-logging` can run the `ON CLUSTER` DDL the +schema migrations use. The helper passes the password to `clickhouse-client` over +standard input, so it never appears in a command line or an audit log: + +```bash +# on the cluster +ch() { # ch — runs clickhouse-client in the first replica + local user=admin; [ "$1" = logging ] && user=platform-logging + kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get secret cpaas-clickhouse-passwords \ + -o jsonpath="{.data.$1}" | base64 -d | \ + kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system exec -i chi-cpaas-clickhouse-replicated-0-0-0 -c clickhouse -- \ + bash -c 'read -r p; exec clickhouse-client -u "$1" --password "$p" -q "$2"' ch "$user" "$2" +} +ch admin "CREATE DATABASE IF NOT EXISTS observability ON CLUSTER 'replicated'" +ch admin "SELECT cluster, shard_num, replica_num, host_name FROM system.clusters WHERE cluster = 'replicated'" +ch logging "CREATE TABLE observability.probe ON CLUSTER 'replicated' (ts DateTime) ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/observability/probe', '{replica}') ORDER BY ts" +ch logging "DROP TABLE observability.probe ON CLUSTER 'replicated'" +``` + +The cluster query returns one row per replica. A `KEEPER_EXCEPTION` means the +Keeper quorum is not serving; `Not enough privileges` names a grant missing from +`73`. Fix the cause and repeat the command — nothing else needs to change +([Step 4: create the database](https://docs.alauda.io/clickhouse/4.4/how_to/deploy_as_logging_storage.html#step-4-create-the-database)). + +**4. Log Storage Management.** The logging documentation describes only a console +installation for this operator, so it remains a console step. In **Marketplace** > +**Cluster Plugins**, select the cluster, install **Alauda Container Platform Log +Storage Management**, and in the advanced configuration set the node selector +`node-role.kubernetes.io/infra` (empty value) and a `NoSchedule` toleration for the +key `node-role.kubernetes.io/infra`. The ingest and query workloads it derives +inherit this placement +([Planning Infra Nodes for Logging Storage](https://docs.alauda.io/logging-service/4.4/how_to/infra_nodes.html)). + +```bash +# on the cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get deployment log-storage-operator -o wide # READY 1/1 +``` + +**5. Connection Secrets.** The operator reads two Secrets by their fixed names. +They contain credentials, so they are created, not applied; both passwords are +read from the Secrets generated in steps 2 and 3. The ClickHouse password is the +plain-text value, not the digest. + +```bash +# on the cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system create secret generic platform-default-mq-conn \ + --from-literal=bootstrap=cpaas-kafka-kafka-bootstrap.cpaas-system.svc:9092 \ + --from-literal=kafkaClusterName=cpaas-kafka \ + --from-literal=username=platform-logging \ + --from-file=password=<(kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system \ + get secret platform-logging-password -o jsonpath='{.data.password}' | base64 -d) \ + --from-literal=sasl_mechanism=SCRAM-SHA-512 \ + --from-literal=topics.log=ALAUDA_LOG_TOPIC \ + --from-literal=topics.event=ALAUDA_EVENT_TOPIC \ + --from-literal=topics.audit=ALAUDA_AUDIT_TOPIC +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system create secret generic platform-default-ch-conn \ + --from-literal=endpoint=http://clickhouse-cpaas-clickhouse.cpaas-system.svc:8123 \ + --from-literal=cluster=replicated \ + --from-literal=database=observability \ + --from-literal=username=platform-logging \ + --from-file=password=<(kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system \ + get secret cpaas-clickhouse-passwords -o jsonpath='{.data.logging}' | base64 -d) +``` + +**6. PlatformLogForward.** The singleton `platform-default` builds the data path. +Shards and replicas must equal the ClickHouse topology; two ingest and two query +replicas match every multi-node ClickHouse profile on the +[capacity page](https://docs.alauda.io/logging-service/4.4/architecture/capacity_planning.html), +and the query-tier resources are the product's per-Pod baseline. + +`/74-platform-log-forward.yaml` — apply on the cluster: + +```yaml +apiVersion: log.alauda.io/v1alpha1 +kind: PlatformLogForward +metadata: + name: platform-default +spec: + output: + type: clickhouse + externalStorage: + type: clickhouse + shards: ${CLICKHOUSE_SHARDS} + replicas: ${CLICKHOUSE_REPLICAS} + secretRef: + name: platform-default-ch-conn + namespace: cpaas-system + externalMessageQueue: + type: kafka + secretRef: + name: platform-default-mq-conn + namespace: cpaas-system + aggregateVector: + replicas: 2 + razor: + replicas: 2 + resources: + limits: + cpu: "4" + memory: 2Gi + requests: + cpu: "1" + memory: 512Mi +``` + +```bash +# on the cluster +apply_rendered wl-prod.env wl-prod/74-platform-log-forward.yaml "${CLUSTER_KUBECONFIG}" +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get platformlogforward platform-default \ + -o jsonpath='{"phase="}{.status.phase}{" ready="}{.status.conditions[?(@.type=="Ready")].status}{"\n"}' # Ready / True +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get pods -l 'app in (aggregate-vector,razor)' -o wide +``` + +**7. Log Collector.** The collector points at the cluster itself and keeps its +working data under `/var/cpaas/data/nevermore`, which cannot be changed after +installation. It is a DaemonSet that must cover every node, so it gets no node +selector. + +`/75-logagent.yaml` — apply on `global`: + +```yaml +apiVersion: cluster.alauda.io/v1alpha1 +kind: ModuleInfo +metadata: + annotations: + cpaas.io/display-name: logagent + cpaas.io/module-name: '{"en": "Alauda Container Platform Log Collector", "zh": "Alauda Container Platform Log Collector"}' + labels: + cpaas.io/cluster-name: ${CLUSTER_NAME} + cpaas.io/module-name: logagent + cpaas.io/module-type: plugin + cpaas.io/product: Platform-Center + name: ${CLUSTER_NAME}-logagent +spec: + version: ${LOGAGENT_VERSION} + config: + crossClusterDependency: + logstorage: ${CLUSTER_NAME} + dataSource: + audit: true + event: true + kubernetes: false + platform: false + system: true + workload: true + components: + nevermore: + storageSpec: + dataDir: /var/cpaas/data/nevermore + storage: + type: Standard +``` + +```bash +# on global +apply_rendered wl-prod.env wl-prod/75-logagent.yaml "${GLOBAL_KUBECONFIG}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get moduleinfo -l cpaas.io/module-name=logagent # STATUS Running +# on the cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n cpaas-system get pods -l app=nevermore -o wide +``` + +**8. Log Essentials** — `global` only, on each side; its name and cluster label +are fixed by the product. + +`global-primary/76-log-api.yaml` (and `global-standby/76-log-api.yaml`) — apply on +`global`: + +```yaml +apiVersion: cluster.alauda.io/v1alpha1 +kind: ModuleInfo +metadata: + annotations: + cpaas.io/display-name: log-api + cpaas.io/module-name: '{"en": "Alauda Container Platform Log Essentials", "zh": "Alauda Container Platform Log Essentials"}' + labels: + cpaas.io/cluster-name: global + cpaas.io/module-name: log-api + cpaas.io/module-type: plugin + cpaas.io/product: Platform-Center + name: global-log-api +spec: + version: ${LOG_API_VERSION} +``` + +```bash +# on global (shown for the primary; repeat with global-standby.env on the standby) +apply_rendered global-primary.env global-primary/76-log-api.yaml "${GLOBAL_KUBECONFIG}" +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get moduleinfo -l cpaas.io/module-name=log-api # STATUS Running +``` + +Then walk the product's seven-step +[post-install verification](https://docs.alauda.io/logging-service/4.4/install/post-install-verification.html), +which ends with a synthetic log queried back end to end. + +Dedicated pools: change `infra` to `log` in every logging placement — the +operator `Subscription` (`70`) and the Kafka operator patch, the four Kafka +tolerations (`controller`, `kafka`, `entityOperator`, `kafkaExporter` in `71`), the +toleration and node affinity of Keeper (`72`) and ClickHouse (`73`), and the +step 4 console placement. For example in `71`: + +```diff + template: + pod: + tolerations: +- - key: node-role.kubernetes.io/infra ++ - key: node-role.kubernetes.io/log + operator: Exists + effect: NoSchedule +``` + +#### DaemonSet coverage + +After every pool is up and every plugin is installed, compare each DaemonSet with +the node count. A DaemonSet that misses a tainted pool fails silently — no error, +just no Pods, so those nodes have no log collection or node metrics. + +```bash +# on the cluster +nodes=$(kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes --no-headers | wc -l) +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get ds -A --no-headers \ + -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,DESIRED:.status.desiredNumberScheduled,READY:.status.numberReady | \ + awk -v n="$nodes" '$3 < n || $4 < $3' +``` + +Every line printed needs an explanation. Expected subsets are the MetalLB speaker +(ingress pool only) and DaemonSets that run on control-plane nodes only. The log +collector, node exporter, CNI and machine configuration daemon must cover every +node; report a gap to product support instead of adding tolerations by hand. + +### Configure machines + +Configure machines right after the Machine Configuration plugin on each cluster, +always in this order: the node disruption policy first, the `MachineConfig` +objects second. A systemd unit change without a policy falls back to the default +action, which drains and reboots every node of the pool, and a file change +without a policy is written to disk without restarting the service, so it does +not take effect +([Node Disruption Policies](https://docs.alauda.io/immutable-infra/1.0/machine-configuration/node_disruption.html)). + +The standard items are the time zone, NTP with the customer's internal source, and +the `boot` user's SSH key. Each `MachineConfig` exists twice, once per default +pool: the `worker` pool and the `master` pool (the control-plane nodes) each take +the objects labeled with their role, and the copies differ only in name and role +label. The `boot` key should be the same key that the provider manifests install +at node creation, so that the two never disagree. + +#### Node disruption policy + +`cluster` is a singleton created by the operator. `kubectl apply` replaces its +`files` and `units` lists as a whole: on a fresh cluster they are empty, but if +they already contain entries, add those entries to this file before applying. + +`/51-node-disruption-policy.yaml` — apply on the cluster: + +```yaml +apiVersion: machineconfiguration.alauda.io/v1alpha1 +kind: MachineConfiguration +metadata: + name: cluster +spec: + nodeDisruptionPolicy: + files: + - path: /etc/chrony.conf + actions: + - type: DaemonReload + - type: Restart + restart: + serviceName: chronyd.service + units: + - name: set-timezone.service + actions: + - type: DaemonReload + - type: Restart + restart: + serviceName: set-timezone.service + sshkey: + actions: + - type: None +``` + +`chronyd` does not support a reload, so it is restarted. + +```bash +# on the cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get machineconfiguration cluster -o jsonpath='{.spec.nodeDisruptionPolicy}'; echo +apply_rendered wl-prod.env wl-prod/51-node-disruption-policy.yaml "${CLUSTER_KUBECONFIG}" +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get machineconfiguration cluster \ + -o jsonpath='{.status.nodeDisruptionPolicyStatus}' | grep -o -e set-timezone.service -e /etc/chrony.conf +``` + +Continue only when both names are printed. Record the boot IDs now, to prove later +that no node rebooted: + +```bash +# on the cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes \ + -o custom-columns=NAME:.metadata.name,BOOT_ID:.status.nodeInfo.bootID > boot-ids-before.txt +``` + +#### Set the time zone + +`/etc/localtime` is a symbolic link, which machine configuration cannot manage, so +a oneshot unit runs `timedatectl set-timezone` +([Setting the Node Time Zone](https://docs.alauda.io/immutable-infra/1.0/machine-configuration/timezone.html)). + +`/52-timezone.yaml` — apply on the cluster: + +```yaml +apiVersion: machineconfiguration.alauda.io/v1alpha1 +kind: MachineConfig +metadata: + name: 99-worker-timezone + labels: + machineconfiguration.alauda.io/role: worker +spec: + config: + ignition: + version: 3.4.0 + systemd: + units: + - name: set-timezone.service + enabled: true + contents: | + [Unit] + Description=Set node timezone + After=multi-user.target + + [Service] + Type=oneshot + ExecStart=/usr/bin/timedatectl set-timezone ${TIMEZONE} + RemainAfterExit=yes + + [Install] + WantedBy=multi-user.target +--- +apiVersion: machineconfiguration.alauda.io/v1alpha1 +kind: MachineConfig +metadata: + name: 99-master-timezone + labels: + machineconfiguration.alauda.io/role: master +spec: + config: + ignition: + version: 3.4.0 + systemd: + units: + - name: set-timezone.service + enabled: true + contents: | + [Unit] + Description=Set node timezone + After=multi-user.target + + [Service] + Type=oneshot + ExecStart=/usr/bin/timedatectl set-timezone ${TIMEZONE} + RemainAfterExit=yes + + [Install] + WantedBy=multi-user.target +``` + +#### Set NTP + +The `chrony.conf` content is generated from `NTP_SERVERS` and stored base64-encoded +in `site.env`, so every cluster receives the identical file +([Configuring the Chrony Time Service](https://docs.alauda.io/immutable-infra/1.0/machine-configuration/managing.html#configuring-the-chrony-time-service)). +Run this once per delivery, and again only when the NTP servers change: + +```bash +# on the delivery host, in the delivery directory +. ./site.env +{ + for server in ${NTP_SERVERS}; do printf 'server %s iburst\n' "$server"; done + printf 'makestep 1.0 3\nrtcsync\nlogdir /var/log/chrony\n' +} > chrony.conf +sed -i '/^CHRONY_CONF_B64=/d' site.env +printf "CHRONY_CONF_B64='%s'\n" "$(base64 -w0 chrony.conf)" >> site.env +``` + +`/53-chrony.yaml` — apply on the cluster: + +```yaml +apiVersion: machineconfiguration.alauda.io/v1alpha1 +kind: MachineConfig +metadata: + name: 99-worker-chrony + labels: + machineconfiguration.alauda.io/role: worker +spec: + config: + ignition: + version: 3.4.0 + storage: + files: + - path: /etc/chrony.conf + mode: 0o644 + contents: + source: 'data:text/plain;base64,${CHRONY_CONF_B64}' +--- +apiVersion: machineconfiguration.alauda.io/v1alpha1 +kind: MachineConfig +metadata: + name: 99-master-chrony + labels: + machineconfiguration.alauda.io/role: master +spec: + config: + ignition: + version: 3.4.0 + storage: + files: + - path: /etc/chrony.conf + mode: 0o644 + contents: + source: 'data:text/plain;base64,${CHRONY_CONF_B64}' +``` + +#### Set the boot user's SSH key + +Machine configuration manages SSH keys for the `boot` user only and does not +create users; the provider manifests create the `boot` user at node creation +(their `KubeadmControlPlane` and `KubeadmConfigTemplate` user entries). SSH key +changes apply without disruption. + +`/54-boot-ssh-key.yaml` — apply on the cluster: + +```yaml +apiVersion: machineconfiguration.alauda.io/v1alpha1 +kind: MachineConfig +metadata: + name: 99-worker-ssh + labels: + machineconfiguration.alauda.io/role: worker +spec: + config: + ignition: + version: 3.4.0 + passwd: + users: + - user: boot + sshAuthorizedKeys: + - "${BOOT_SSH_PUBLIC_KEY}" +--- +apiVersion: machineconfiguration.alauda.io/v1alpha1 +kind: MachineConfig +metadata: + name: 99-master-ssh + labels: + machineconfiguration.alauda.io/role: master +spec: + config: + ignition: + version: 3.4.0 + passwd: + users: + - user: boot + sshAuthorizedKeys: + - "${BOOT_SSH_PUBLIC_KEY}" +``` + +#### Verify the machine configuration + +```bash +# on the cluster +for f in 52-timezone 53-chrony 54-boot-ssh-key; do + apply_rendered wl-prod.env "wl-prod/$f.yaml" "${CLUSTER_KUBECONFIG}" || break +done +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get machineconfigpool # wait for UPDATED=True, DEGRADED=False +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get node -o custom-columns=NODE:.metadata.name,\ +DESIRED:.metadata.annotations."machineconfiguration\.alauda\.io/desiredConfig",\ +CURRENT:.metadata.annotations."machineconfiguration\.alauda\.io/currentConfig",\ +STATE:.metadata.annotations."machineconfiguration\.alauda\.io/state" # DESIRED = CURRENT, STATE Done +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes \ + -o custom-columns=NAME:.metadata.name,BOOT_ID:.status.nodeInfo.bootID | diff boot-ids-before.txt - \ + && echo "no node rebooted" +``` + +Spot-check one node per pool as the `boot` user with the delivered key: + +```bash +# on a node +timedatectl # Time zone = TIMEZONE, System clock synchronized: yes +chronyc sources -v # the customer's NTP servers, one selected (*) +systemctl status set-timezone.service # active (exited) +``` + +A node whose files are changed by hand is marked `Degraded` and stops receiving +configuration; restore it as described in +[Managing Node Configuration](https://docs.alauda.io/immutable-infra/1.0/machine-configuration/managing.html) +instead of editing more files. + +### Size the clusters + +Derive every size from the product capacity pages instead of copying numbers from +another delivery. Collect the inputs first: the number of managed clusters, the +worker node and Pod counts per cluster, the sustained log volume (logs per second +at peak) and its retention, and the metrics retention. + +1. **`global` control plane** — choose the tier for the number of managed + clusters in + [Evaluating Resources for Global Cluster](https://docs.alauda.io/container_platform/4.4/configure/scalability/evaluating_global.html) + and apply its vertical and horizontal scaling rules. Both sides of a DR pair + are sized identically: after a failover the standby carries the full load. +2. **Workload control plane** — size by worker count and object density with + [Evaluating Resources for Workload Cluster](https://docs.alauda.io/container_platform/4.4/configure/scalability/evaluating_workload.html), + and keep the headroom it recommends for one control-plane node being offline. +3. **Monitoring** — pick the VictoriaMetrics scale whose worker-node and Pod + counts cover the cluster in + [Monitor Component Capacity Planning](https://docs.alauda.io/container_platform/4.4/observability/monitor/architecture/capacity_planning.html) + (`MONITOR_SIZE`). The page's disk figures are for a 7-day retention; scale them + proportionally to `MONITOR_RETENTION_DAYS` and add headroom + (`MONITOR_STORAGE_GI`). +4. **Logging** — pick the ClickHouse profile whose throughput covers the peak log + rate in + [Log Component Capacity Planning](https://docs.alauda.io/logging-service/4.4/architecture/capacity_planning.html) + (`CLICKHOUSE_SHARDS`, `CLICKHOUSE_REPLICAS`) and take the Kafka broker count + from the same profile. Size Kafka volumes as retention × daily volume × + replication factor (`KAFKA_STORAGE_SIZE`) and ClickHouse volumes from the + customer's measured daily log volume and retention (`CLICKHOUSE_STORAGE_SIZE`). + Throughput above the largest profile is a design change, not a bigger number. +5. **Infra pool (or dedicated pools)** — add up the CPU and memory limits of every + component placed on the pool (the plugin tables of steps 3 and 4, plus the + ingress controllers) and size the nodes so that the pool still fits with one + node down, within the utilization ranges of the evaluation pages. Pods are + scheduled by their requests, so also check that the free requests of each + logging node cover one ClickHouse replica and one Keeper member (`73`, `72`). + The three-node HA profiles set the minimum node count of the logging pool. +6. **`/var/cpaas` per node** — platform state plus the data the node hosts: its + vmstorage share and the collector working data, plus its Kafka, Keeper and + ClickHouse volumes where their StorageClass provisions on this disk. Keep the disks on + the baseline the capacity pages assume (SSD, about 6000 IOPS and 250 MB/s). +7. **Bootstrap node** — at least 4 cores, 8 GB memory and 300 GB disk on x86_64 + ([Bootstrap Node Requirements](https://docs.alauda.io/immutable-infra/1.0/global/install.html#bootstrap-host-requirements)). + +The capacity tables are laboratory references: provision above them, and +recalculate and validate anything beyond the documented tiers rather than +extrapolating. + +### Accept the delivery + +Accept against evidence, not against "the installer finished". Run the checks on +every cluster (`load_env .env` first) and archive their output. + +**Nodes, pools and endpoint** + +```bash +# on the cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes -o wide # all Ready, planned counts and IPs +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes \ + -o custom-columns=NAME:.metadata.name,VERSION:.status.nodeInfo.kubeletVersion,PROVIDER_ID:.spec.providerID,TAINTS:.spec.taints[*].key +echo "expected version: ${K8S_VERSION}" # every node reports it; no empty providerID +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get nodes -l node-role.kubernetes.io/infra +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" config view --minify -o jsonpath='{.clusters[0].cluster.server}'; echo +echo "expected endpoint: https://${CONTROL_PLANE_VIP}:6443" # an IP address, never a domain +curl -kfsS "https://${CONTROL_PLANE_VIP}:6443/healthz"; echo # ok +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" -n kube-system get cm kubeadm-config \ + -o jsonpath='{.data.ClusterConfiguration}' | grep -E 'podSubnet|serviceSubnet' +echo "expected: ${CLUSTER_CIDR} ${SERVICE_CIDR}" +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get subnet join -o jsonpath='{.spec.cidrBlock}'; echo +echo "expected: ${KUBE_OVN_JOIN_CIDR}" +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" -n cpaas-system get machines \ + -l cluster.x-k8s.io/cluster-name="${CLUSTER_NAME}" # every Machine Running +``` + +**Platform access, certificates and registry** (on the active `global`) + +```bash +# on the delivery host +getent hosts "${PLATFORM_HOST}" # the primary global VIP +echo | openssl s_client -connect "${PLATFORM_HOST}:443" -servername "${PLATFORM_HOST}" 2>/dev/null | \ + openssl x509 -noout -subject -enddate -ext subjectAltName # SAN contains PLATFORM_HOST +curl -k -s -o /dev/null -w '%{http_code}\n' "https://${REGISTRY_DOMAIN}/v2/" # 200 or 401: the registry answers +# on the cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get events -A | grep -E 'ErrImagePull|ImagePullBackOff' # nothing +``` + +**Plugins and placement** + +```bash +# on global +kubectl --kubeconfig "${GLOBAL_KUBECONFIG}" get moduleinfo -l cpaas.io/cluster-name="${CLUSTER_NAME}" # all Running +# on the cluster +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get pods -A -o wide | \ + grep -E 'metallb|ingress-nginx|vmstorage|vminsert|vmselect|vmagent|vmalert|alertmanager|cpaas-kafka|keeper|clickhouse|aggregate-vector|razor|log-storage-operator' +kubectl --kubeconfig "${CLUSTER_KUBECONFIG}" get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded +``` + +Every listed Pod runs on a node of its planned pool (see the placement table in +[Node pools, labels and taints](#node-pools-labels-and-taints)), the DaemonSet +coverage check prints no unexplained line, the ingress VIP answers, and the +logging post-install verification passed. + +**Machine configuration** — every `MachineConfigPool` reports `UPDATED=True` and +`DEGRADED=False`, every node `STATE Done`, no unplanned reboot, and the node +spot-checks of [Configure machines](#configure-machines) pass. + +**Disaster recovery** — both sides passed their installation verification, the +standby's plugins are `Running` on the standby, and synchronization reports zero +missed and zero surplus keys. Run the check on a control-plane node of the standby +(the Service address is cluster-internal): + +```bash +# on a control-plane node of the standby global +mirror_host=$(sudo kubectl -n cpaas-system get svc etcd-sync-monitor -o jsonpath='{.spec.clusterIP}') +case "$mirror_host" in *:*) mirror_host="[$mirror_host]" ;; esac # IPv6 Service address +curl -g "http://$mirror_host/check" +``` + +Run the failover drill per +[Failover](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html#failover-procedure) +with the customer, and after it confirm on the new active side that every workload +`Cluster` and `Machine` is intact and that the provider credential Secrets exist. +After a failover, `GLOBAL_KUBECONFIG` in every workload cluster file must point at +the new active side. + +**Credentials and archive** + +Render every manifest into `rendered/` for the archive, then scan the whole +delivery directory for credentials. The scan matches a value after `password:` or +`password=`, a `*_TOKEN=` assignment, an `aescbc` encryption key and a private +key block: + +```bash +# on the delivery host, in the delivery directory +for d in global-primary global-standby wl-*/; do + d=${d%/}; [ -d "$d" ] && [ -f "$d.env" ] || continue + mkdir -p "rendered/$d" + for m in "$d"/*.yaml; do render "$d.env" "$m" > "rendered/$m"; done +done +grep -rnIiE 'aescbc|password[[:space:]]*[:=][[:space:]]*[^[:space:]]|_TOKEN=|BEGIN [A-Z ]*PRIVATE KEY' . # nothing +``` + +Archive the parameter files, the rendered manifests and the output of every check +above. Record Secret names and their custodians, never their values; kubeconfigs, +tokens and private keys stay out of the archive. If the delivery deviates from +this design, stop and resolve the deviation with the platform team before +continuing — a deviation that is cheap to fix at the design stage becomes a +re-installation later. + +## Related Information + +- [How to Deliver ACP on Huawei DCS with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_on_Huawei_DCS_with_Immutable_Infrastructure.md) +- [How to Deliver ACP on Huawei Cloud Stack with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_on_Huawei_Cloud_Stack_with_Immutable_Infrastructure.md) +- [How to Deliver ACP on Bare Metal with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_on_Bare_Metal_with_Immutable_Infrastructure.md) +- [How to Deliver ACP on VMware vSphere with Immutable Infrastructure](/solutions/acp/How_to_Deliver_ACP_on_VMware_vSphere_with_Immutable_Infrastructure.md) +- [Installing the global Cluster (Immutable Infrastructure)](https://docs.alauda.io/immutable-infra/1.0/global/install.html) +- [Global Cluster Disaster Recovery (Immutable Infrastructure)](https://docs.alauda.io/immutable-infra/1.0/global/disaster_recovery.html) +- [Creating Clusters](https://docs.alauda.io/immutable-infra/1.0/create-cluster/index.html) +- [Plan the Control Plane Endpoint](https://docs.alauda.io/immutable-infra/1.0/how-to/control-plane-endpoint.html) +- [Machine Configuration Overview](https://docs.alauda.io/immutable-infra/1.0/machine-configuration/overview.html) +- [Cluster Plugin](https://docs.alauda.io/container_platform/4.4/extend/cluster_plugin.html#installing-via-yaml) +- [Monitoring Installation](https://docs.alauda.io/container_platform/4.4/observability/monitor/install_monitor.html) +- [Logging Installation](https://docs.alauda.io/logging-service/4.4/install/index.html)