Title: L2STS links down since ~14:50 ET 11/03, re-adding them fails

Hi,

Thank you for the detailed report and timeline. It helped us find the problem quickly.

I apologize for the inconvenience.

What happened

– 10/31 renew: A bug in how lease end times were compared made your network services fail to extend, even though the VMs they’re attached to extended fine. This is fixed, and should not impact any future extend requests.
– 11/03 outage: Because of that failed renew, the network AM’s copy of your slice still had the old 10/31 expiry. An automated cleanup job then treated the slice as expired and removed it from the network AM’s records, even though your re-created services were still active. From that point the network AM couldn’t manage network services for this slice. That’s why all six links stopped working, and why every attempt to delete or re-add them failed with Slice Not Found. Nothing you did caused this. We’re also fixing the cleanup job so it can’t remove a slice that still has active services.

Your VMs are not affected. All 19 nodes are still running. The network interfaces belong to the VMs, not to the network services, so your in-VM IP addresses and FRR configuration will carry over once the networks are re-created.

We have resynced the network AM for your slice and clearing the stale VLAN configuration on the RUTG, PRIN, WASH and STAR switches.

Please use the code below to re-add network services. It deletes any leftover network services, then re-creates net1-4 and the six router links on the same interfaces and subnets as before, and runs a quick ping check.

# readd_networks.py — re-create net1-4 and the 6 router links in OSPF_Routing_Param4
from ipaddress import IPv4Network
from fabrictestbed_extensions.fablib.fablib import FablibManager

SLICE_NAME = “OSPF_Routing_Param4”

# name -> (type, [node interface names], subnet or None)
NETWORKS = {
“net1”: (“L2Bridge”, [“router1-nic_local-p1”, “rutg1-nic_local-p1”, “rutg2-nic_local-p1”,
“rutg3-nic_local-p1”, “rutg4-nic_local-p1”, “rutg5-nic_local-p1”], None),
“net2”: (“L2Bridge”, [“router2-nic_local-p1”, “prin1-nic_local-p1”, “prin2-nic_local-p1”,
“prin3-nic_local-p1”], None),
“net3”: (“L2Bridge”, [“router3-nic_local-p1”, “wash1-nic_local-p1”, “wash2-nic_local-p1”,
“wash3-nic_local-p1”, “wash4-nic_local-p1”], None),
“net4”: (“L2Bridge”, [“router4-nic_local-p1”, “star1-nic_local-p1”, “star2-nic_local-p1”,
“star3-nic_local-p1”], None),
“link_r1_r2”: (“L2STS”, [“router1-r1-r2-p1”, “router2-r2-r1-p1”], “192.168.101.0/24”),
“link_r1_r3”: (“L2STS”, [“router1-r1-r3-p1”, “router3-r3-r1-p1”], “192.168.102.0/24”),
“link_r1_r4”: (“L2STS”, [“router1-r1-r4-p1”, “router4-r4-r1-p1”], “192.168.103.0/24”),
“link_r2_r3”: (“L2STS”, [“router2-r2-r3-p1”, “router3-r3-r2-p1”], “192.168.104.0/24”),
“link_r2_r4”: (“L2STS”, [“router2-r2-r4-p1”, “router4-r4-r2-p1”], “192.168.105.0/24”),
“link_r3_r4”: (“L2STS”, [“router3-r3-r4-p1”, “router4-r4-r3-p1”], “192.168.106.0/24”),
}

fablib = FablibManager()

# ———- Step 1: remove any leftover network services ———-
slice = fablib.get_slice(name=SLICE_NAME)
print(“State before:”, slice.get_state())
existing = slice.get_network_services()
print(“Deleting:”, [n.get_name() for n in existing])
for net in existing:
net.delete()

if existing:
# If this times out, do NOT re-run submit(). Check
# fablib.get_slice(name=SLICE_NAME).get_state() and wait for StableOK/ModifyOK.
slice.submit()

slice = fablib.get_slice(name=SLICE_NAME)
left = [n.get_name() for n in slice.get_network_services()]
assert not left, f”Network services still present, please stop and contact us: {left}”
print(“All network services removed; state:”, slice.get_state())

# ———- Step 2: re-create the networks on the same interfaces ———-
ifaces = {i.get_name(): i for n in slice.get_nodes() for i in n.get_interfaces()}

for name, (ns_type, if_names, subnet) in NETWORKS.items():
missing = [x for x in if_names if x not in ifaces]
if missing:
raise RuntimeError(f”{name}: interfaces not found {missing}; available: {sorted(ifaces)}”)
slice.add_l2network(name=name, type=ns_type,
interfaces=[ifaces[x] for x in if_names],
subnet=IPv4Network(subnet) if subnet else None)

slice.submit(wait=True, progress=True)

# ———- Step 3: verify ———-
slice = fablib.get_slice(name=SLICE_NAME)
print(“State after:”, slice.get_state())
for net in slice.get_network_services():
print(f”{net.get_name():12s} {net.get_type()} {net.get_reservation_state()} {net.get_error_message()}”)

# Router-to-router reachability over each link (adjust if your .1/.2 assignment differs)
PEERS = {“router1”: [“192.168.101.2”, “192.168.102.2”, “192.168.103.2”],
“router2”: [“192.168.104.2”, “192.168.105.2”],
“router3”: [“192.168.106.2″]}
for node_name, ips in PEERS.items():
node = slice.get_node(name=node_name)
for ip in ips:
out, _ = node.execute(f”ping -c 3 -W 2 {ip} | tail -2″, quiet=True)
print(f”{node_name} -> {ip}: {out.strip()}”)

A few notes:

– net1-4 are re-created too. Their current copies have the same problem as the links, so re-creating everything at once is the cleanest fix.
– Timeouts: if submit() times out, don’t run it again. Earlier, the retries after timeouts created duplicate requests: your two attempts became four rounds of submissions. Check the slice state and wait for it to settle instead.
– Peer IPs: the ping check assumes .1 is the lower-numbered router on each link. Adjust PEERS if yours are the other way round.
– OSPF: if adjacencies don’t come back on their own within a minute or two, run vtysh -c “clear ip ospf process” in each router container.

If anything in the script fails, send us the output and we’ll take it from there. Sorry for the disruption to your experiment.

Best regards,
Komal