problem
When the last vif is unplugged from the public bridge, the agent deletes the bridge and its parent VLAN interface — even though CloudStack never created them.
BridgeVifDriver.deleteVnetBr() decides whether a bridge is CloudStack-owned solely by matching its name against a regex:
Pattern brNameRegex = Pattern.compile("^br(\\S+)-(\\d+)$");
This is intended to match bridges the agent created via generateVnetBrName() ("br" + pifName + "-" + vnetId). But the pattern matches any similarly-named bridge. There is no provenance tracking — the naming convention is the only ownership test.
A pre-existing, persistently-declared bridge named brbond0-103 (bond bond0, VLAN 103) matches exactly, so the agent treats it as its own and runs:
modifyvlan.sh -o delete -v 103 -p bond0 -b brbond0-103 -d true
which deletes both interfaces:
ip link delete $vlanDev type vlan # → ip link delete bond0.103
ip link set $vlanBr down
ip link delete $vlanBr type bridge # → ip link delete brbond0-103
netplan/networkd do not recreate these without netplan apply or a reboot, so the loss is permanent until manual intervention.
The failure is silent for as long as the agent keeps running. The public NIC is validated only as a one-time startup precondition in LibvirtComputingResource.configure():
if (pifs.get("public") == null) {
LOGGER.error("Failed to get public nic name");
throw new ConfigurationException("Failed to get public nic name");
}
There is no retry and no degraded mode. The running agent never re-checks — it merely logs, once per minute:
WARN [kvm.resource.LibvirtComputingResource] Failed to read the rx_bytes for brbond0-103
from /sys/class/net/brbond0-103/statistics/rx_bytes
java.io.FileNotFoundException: File '/sys/class/net/brbond0-103/statistics/rx_bytes' does not exist
so the host continues to report healthy. The damage only surfaces at the next agent restart — in our case 44 hours later — when the host fails to reconnect.
versions
ACS 4.22.1.0
Ubuntu 24.04
KVM hypervisor, Ubuntu 24.04, systemd-networkd + netplan.
The Public traffic type uses a persistently declared bridge, created at boot by netplan — not by CloudStack:
# /etc/netplan/50-cloud-init.yaml (abridged)
vlans:
bond0.103:
id: 103
link: bond0
mtu: 1500
bridges:
brbond0-103:
interfaces: [bond0.103]
mtu: 1500
parameters: {stp: false, forward-delay: 0}
agent.properties
public.network.device=brbond0-103
Zone config: Public traffic type has kvmnetworklabel=brbond0-103; public IP ranges are vlan://103.
OS / ENVIRONMENT
Ubuntu 24.04, kernel 6.11, systemd-networkd/netplan, bonded 802.3ad uplink
The steps to reproduce the bug
- On a KVM host, persistently declare (netplan/networkd/ifupdown) a VLAN interface
and bridge whose name matches ^br(\S+)-(\d+)$ — e.g. bond0.103 enslaved to
brbond0-103.
- Set public.network.device=brbond0-103 and the zone's Public
kvmnetworklabel=brbond0-103, with a vlan://103 public IP range.
- Start the agent; create an isolated network so a VR is placed on this host with
a nic on the public bridge. Confirm brbond0-103 has the VR vif as its only VM port.
- Destroy that VR (or restart the network so the VR is recreated elsewhere), so the
last vif leaves the bridge.
- Observe: bond0.103 and brbond0-103 are deleted from the running kernel.
- Restart cloudstack-agent.
EXPECTED RESULTS
The agent does not delete network interfaces it did not create. Bridges and VLAN
interfaces that are persistently declared by the host's network configuration are
left in place when the last vif is unplugged.
ACTUAL RESULTS
Step 5 — kernel logs the interfaces being removed (note "unregistering", i.e.
device deletion rather than link-down):
kernel: brbond0-103: port 1(bond0.103) entered disabled state
systemd-networkd: bond0.103: Link DOWN
kernel: bond0.103 (unregistering): left promiscuous mode
systemd-networkd: brbond0-103: Link DOWN
Step 6 — the agent then refuses to start:
ERROR [kvm.resource.LibvirtComputingResource] Failed to get public nic name
ERROR [cloud.agent.AgentShell] Unable to start agent:
javax.naming.ConfigurationException: Failed to get public nic name
at LibvirtComputingResource.configure(LibvirtComputingResource.java:1495)
at com.cloud.agent.Agent.<init>(Agent.java:239)
...
The host stays Disconnected until the interfaces are recreated by hand. Guest VMs
already running are unaffected (libvirt and the data plane are independent of the
agent), which is part of why the fault goes unnoticed.
What to do about it?
Impact is wider than it first appears. The trigger is "the last vif leaves the public bridge", not "the host is idle" — in our case the host had four VMs running throughout, none of them on the public bridge. Since only virtual routers attach to the public network, and VRs are among the most frequently destroyed/recreated VMs (network restart, offering change, VR upgrade, HA event), any host that transiently has no VR on the public network is exposed. The resulting breakage is invisible until the next agent restart, which in practice means it surfaces during a maintenance window.
Why "just name it differently" is not a workaround. The bridge name used at plug time is derived, not read from config — createVnetBr() → generateVnetBrName(_pifs.get(trafficLabel), vNetId) → "br" + "bond0" + "-" + "103". Renaming the declared bridge causes the agent to create a second bridge under the constructed name rather than using the renamed one. For a VLAN-tagged public network there is no configuration that avoids the matching name.
Suggested fixes (in rough order of preference):
Track provenance — only delete bridges the agent itself created (e.g. a marker, or state recorded at creation time), rather than inferring ownership from a name.
Failing that, an opt-out agent property, e.g. network.bridge.externally.managed=brbond0-103,..., consulted by deleteVnetBr() before deletion.
At minimum, never delete the parent VLAN interface. Even where the bridge is plausibly CloudStack's, bond0.103 belongs to the host's network configuration.
Two related observations, filed separately to keep this report focused:
modifyvlan.sh sets no MTU when creating interfaces, so an agent-created VLAN inherits the parent's (9000 on a jumbo-frame bond) rather than the intended value.
The ls /sys/class/net/
/brif "is the bridge in use?" check is not atomic with the deletion, so a concurrent plug can race it.
Happy to test a patch.
problem
When the last vif is unplugged from the public bridge, the agent deletes the bridge and its parent VLAN interface — even though CloudStack never created them.
BridgeVifDriver.deleteVnetBr() decides whether a bridge is CloudStack-owned solely by matching its name against a regex:
Pattern brNameRegex = Pattern.compile("^br(\\S+)-(\\d+)$");This is intended to match bridges the agent created via generateVnetBrName() ("br" + pifName + "-" + vnetId). But the pattern matches any similarly-named bridge. There is no provenance tracking — the naming convention is the only ownership test.
A pre-existing, persistently-declared bridge named brbond0-103 (bond bond0, VLAN 103) matches exactly, so the agent treats it as its own and runs:
modifyvlan.sh -o delete -v 103 -p bond0 -b brbond0-103 -d truewhich deletes both interfaces:
netplan/networkd do not recreate these without netplan apply or a reboot, so the loss is permanent until manual intervention.
The failure is silent for as long as the agent keeps running. The public NIC is validated only as a one-time startup precondition in LibvirtComputingResource.configure():
There is no retry and no degraded mode. The running agent never re-checks — it merely logs, once per minute:
so the host continues to report healthy. The damage only surfaces at the next agent restart — in our case 44 hours later — when the host fails to reconnect.
versions
ACS 4.22.1.0
Ubuntu 24.04
KVM hypervisor, Ubuntu 24.04, systemd-networkd + netplan.
The Public traffic type uses a persistently declared bridge, created at boot by netplan — not by CloudStack:
agent.properties
public.network.device=brbond0-103Zone config: Public traffic type has kvmnetworklabel=brbond0-103; public IP ranges are vlan://103.
OS / ENVIRONMENT
Ubuntu 24.04, kernel 6.11, systemd-networkd/netplan, bonded 802.3ad uplink
The steps to reproduce the bug
and bridge whose name matches ^br(\S+)-(\d+)$ — e.g. bond0.103 enslaved to
brbond0-103.
kvmnetworklabel=brbond0-103, with a vlan://103 public IP range.
a nic on the public bridge. Confirm brbond0-103 has the VR vif as its only VM port.
last vif leaves the bridge.
EXPECTED RESULTS
The agent does not delete network interfaces it did not create. Bridges and VLAN
interfaces that are persistently declared by the host's network configuration are
left in place when the last vif is unplugged.
ACTUAL RESULTS
Step 5 — kernel logs the interfaces being removed (note "unregistering", i.e.
device deletion rather than link-down):
Step 6 — the agent then refuses to start:
The host stays Disconnected until the interfaces are recreated by hand. Guest VMs
already running are unaffected (libvirt and the data plane are independent of the
agent), which is part of why the fault goes unnoticed.
What to do about it?
Impact is wider than it first appears. The trigger is "the last vif leaves the public bridge", not "the host is idle" — in our case the host had four VMs running throughout, none of them on the public bridge. Since only virtual routers attach to the public network, and VRs are among the most frequently destroyed/recreated VMs (network restart, offering change, VR upgrade, HA event), any host that transiently has no VR on the public network is exposed. The resulting breakage is invisible until the next agent restart, which in practice means it surfaces during a maintenance window.
Why "just name it differently" is not a workaround. The bridge name used at plug time is derived, not read from config — createVnetBr() → generateVnetBrName(_pifs.get(trafficLabel), vNetId) → "br" + "bond0" + "-" + "103". Renaming the declared bridge causes the agent to create a second bridge under the constructed name rather than using the renamed one. For a VLAN-tagged public network there is no configuration that avoids the matching name.
Suggested fixes (in rough order of preference):
Track provenance — only delete bridges the agent itself created (e.g. a marker, or state recorded at creation time), rather than inferring ownership from a name.
Failing that, an opt-out agent property, e.g. network.bridge.externally.managed=brbond0-103,..., consulted by deleteVnetBr() before deletion.
At minimum, never delete the parent VLAN interface. Even where the bridge is plausibly CloudStack's, bond0.103 belongs to the host's network configuration.
Two related observations, filed separately to keep this report focused:
modifyvlan.sh sets no MTU when creating interfaces, so an agent-created VLAN inherits the parent's (9000 on a jumbo-frame bond) rather than the intended value.
The ls /sys/class/net/
/brif "is the bridge in use?" check is not atomic with the deletion, so a concurrent plug can race it.
Happy to test a patch.