Forum Replies Created
-
AuthorPosts
-
November 3, 2026 at 6:26 pm in reply to: Title: L2STS links down since ~14:50 ET 11/03, re-adding them fails #10125
Hi,
Thank you for the detailed report and timeline. It helped us find the problem quickly.
I apologize for the inconvenience.
What happened
– 10/31 renew: A bug in how lease end times were compared made your network services fail to extend, even though the VMs they’re attached to extended fine. This is fixed, and should not impact any future extend requests.
– 11/03 outage: Because of that failed renew, the network AM’s copy of your slice still had the old 10/31 expiry. An automated cleanup job then treated the slice as expired and removed it from the network AM’s records, even though your re-created services were still active. From that point the network AM couldn’t manage network services for this slice. That’s why all six links stopped working, and why every attempt to delete or re-add them failed with Slice Not Found. Nothing you did caused this. We’re also fixing the cleanup job so it can’t remove a slice that still has active services.Your VMs are not affected. All 19 nodes are still running. The network interfaces belong to the VMs, not to the network services, so your in-VM IP addresses and FRR configuration will carry over once the networks are re-created.
We have resynced the network AM for your slice and clearing the stale VLAN configuration on the RUTG, PRIN, WASH and STAR switches.
Please use the code below to re-add network services. It deletes any leftover network services, then re-creates net1-4 and the six router links on the same interfaces and subnets as before, and runs a quick ping check.
# readd_networks.py — re-create net1-4 and the 6 router links in OSPF_Routing_Param4
from ipaddress import IPv4Network
from fabrictestbed_extensions.fablib.fablib import FablibManagerSLICE_NAME = “OSPF_Routing_Param4”
# name -> (type, [node interface names], subnet or None)
NETWORKS = {
“net1”: (“L2Bridge”, [“router1-nic_local-p1”, “rutg1-nic_local-p1”, “rutg2-nic_local-p1”,
“rutg3-nic_local-p1”, “rutg4-nic_local-p1”, “rutg5-nic_local-p1”], None),
“net2”: (“L2Bridge”, [“router2-nic_local-p1”, “prin1-nic_local-p1”, “prin2-nic_local-p1”,
“prin3-nic_local-p1”], None),
“net3”: (“L2Bridge”, [“router3-nic_local-p1”, “wash1-nic_local-p1”, “wash2-nic_local-p1”,
“wash3-nic_local-p1”, “wash4-nic_local-p1”], None),
“net4”: (“L2Bridge”, [“router4-nic_local-p1”, “star1-nic_local-p1”, “star2-nic_local-p1”,
“star3-nic_local-p1”], None),
“link_r1_r2”: (“L2STS”, [“router1-r1-r2-p1”, “router2-r2-r1-p1”], “192.168.101.0/24”),
“link_r1_r3”: (“L2STS”, [“router1-r1-r3-p1”, “router3-r3-r1-p1”], “192.168.102.0/24”),
“link_r1_r4”: (“L2STS”, [“router1-r1-r4-p1”, “router4-r4-r1-p1”], “192.168.103.0/24”),
“link_r2_r3”: (“L2STS”, [“router2-r2-r3-p1”, “router3-r3-r2-p1”], “192.168.104.0/24”),
“link_r2_r4”: (“L2STS”, [“router2-r2-r4-p1”, “router4-r4-r2-p1”], “192.168.105.0/24”),
“link_r3_r4”: (“L2STS”, [“router3-r3-r4-p1”, “router4-r4-r3-p1”], “192.168.106.0/24”),
}fablib = FablibManager()
# ———- Step 1: remove any leftover network services ———-
slice = fablib.get_slice(name=SLICE_NAME)
print(“State before:”, slice.get_state())
existing = slice.get_network_services()
print(“Deleting:”, [n.get_name() for n in existing])
for net in existing:
net.delete()if existing:
# If this times out, do NOT re-run submit(). Check
# fablib.get_slice(name=SLICE_NAME).get_state() and wait for StableOK/ModifyOK.
slice.submit()slice = fablib.get_slice(name=SLICE_NAME)
left = [n.get_name() for n in slice.get_network_services()]
assert not left, f”Network services still present, please stop and contact us: {left}”
print(“All network services removed; state:”, slice.get_state())# ———- Step 2: re-create the networks on the same interfaces ———-
ifaces = {i.get_name(): i for n in slice.get_nodes() for i in n.get_interfaces()}for name, (ns_type, if_names, subnet) in NETWORKS.items():
missing = [x for x in if_names if x not in ifaces]
if missing:
raise RuntimeError(f”{name}: interfaces not found {missing}; available: {sorted(ifaces)}”)
slice.add_l2network(name=name, type=ns_type,
interfaces=[ifaces[x] for x in if_names],
subnet=IPv4Network(subnet) if subnet else None)slice.submit(wait=True, progress=True)
# ———- Step 3: verify ———-
slice = fablib.get_slice(name=SLICE_NAME)
print(“State after:”, slice.get_state())
for net in slice.get_network_services():
print(f”{net.get_name():12s} {net.get_type()} {net.get_reservation_state()} {net.get_error_message()}”)# Router-to-router reachability over each link (adjust if your .1/.2 assignment differs)
PEERS = {“router1”: [“192.168.101.2”, “192.168.102.2”, “192.168.103.2”],
“router2”: [“192.168.104.2”, “192.168.105.2”],
“router3”: [“192.168.106.2″]}
for node_name, ips in PEERS.items():
node = slice.get_node(name=node_name)
for ip in ips:
out, _ = node.execute(f”ping -c 3 -W 2 {ip} | tail -2″, quiet=True)
print(f”{node_name} -> {ip}: {out.strip()}”)A few notes:
– net1-4 are re-created too. Their current copies have the same problem as the links, so re-creating everything at once is the cleanest fix.
– Timeouts: if submit() times out, don’t run it again. Earlier, the retries after timeouts created duplicate requests: your two attempts became four rounds of submissions. Check the slice state and wait for it to settle instead.
– Peer IPs: the ping check assumes .1 is the lower-numbered router on each link. Adjust PEERS if yours are the other way round.
– OSPF: if adjacencies don’t come back on their own within a minute or two, run vtysh -c “clear ip ospf process” in each router container.If anything in the script fails, send us the output and we’ll take it from there. Sorry for the disruption to your experiment.
Best regards,
KomalOctober 30, 2026 at 9:01 am in reply to: L2Bridge link fails at STAR with “Interface already used for L2VPN” #10115Hi Haribabu,
Could you please try creating your slice again?
Best,
Komal
October 30, 2026 at 8:54 am in reply to: Portal and FABLib cannot list slices — HTTP 502 Bad Gateway #10114An unexpected infrastructure outage yesterday caused these issues. More details are available in the outage announcement.
Testbed services have been restored, and your slices should now be visible.
Best,
KomalDear Experimenters,
Control Framework services have been restored, and you can now resume slice operations.
We apologize for the inconvenience.
Best,
KomalLooking at it now. May i please ask which JH container option are you running?
I mean which option did you select when you logged in to JH?
Best,
Komal
- WASH may have enough resources in aggregate, but individual hosts may lack the capacity needed for your requested VMs, resulting in the
Insufficient Resourceserror. - Resources associated with a Closing slice are released, as they are for a Dead slice. Delayed responses to resource closure requests can sometimes leave a slice stuck in the Closing state.
- This may have been a transient error. The slice looks healthy, and all requested resources are now Active.
Best,
Komal
- WASH may have enough resources in aggregate, but individual hosts may lack the capacity needed for your requested VMs, resulting in the
Insufficient Resourceserror. - Resources associated with a Closing slice are released, as they are for a Dead slice. Delayed responses to resource closure requests can sometimes leave a slice stuck in the Closing state.
- This may have been a transient error. The slice looks healthy, and all requested resources are now Active.
Best,
Komal
October 19, 2026 at 10:08 am in reply to: FIU Tofino switch refused (Insufficient resources: [unit]) #10092Hi,
- Your switch request was rejected because the resource was unavailable, as indicated by the error
Insufficient resources: [unit]. We currently have only five Tofino switches, and demand is high. Please check the Resource Calendar in LoomAI for future availability. Because FABRIC is a distributed system, another request may have acquired the switch between the availability check and the processing of your request. I have verified that another slice holds the switch and was created at the similar time when your slice requests were triggered. - All resources associated with slices in the Closing or Dead state are released.
Best,
KomalHi Vakhob,
If you run into a token error, please restart your JH container via
File -> Hub Control Panel -> Stop My Server followed by Start My Server.
Also, to avoid the Unique name error, please restart the Notebook kernel.
If not using already, please consider using the new recommended way of using FABRIC via loomai.net.
Best,
Komal
Hi Romaric,
We may not have the on Portal yet. But if you share which publication needs to be associated with the corresponding project we can update that for now. We will also work to address this on the portal.
Best,
Komal
Hi Ivan,
Could you please share your slice ID?
Best,
Komal
Hi Stephen,
There was a mismatch in the code for
configurefunction and the actual BFB image on the host. I pushed a fix for this and updated the notebook too. The notebook should work now.Thank you for sharing your observation and helping us address the issue. Appreciate it!
Best,
Komal
I’m glad to hear the issue has been resolved! Thank you for sharing the error message. We’ll work on making it clearer and more helpful.
Best,
KomalHi Garegin,
Could you please share the error they are encountering?
I verified that the
configure_and_validate.ipynbnotebook works without any issues with the latest API.We also recommend using loomai.net as the new interface for accessing FABRIC. LoomAI is an AI-assisted graphical interface that also provides access to the existing JupyterHub environment.
Also, please try File -> Hub Control Panel -> Stop My Server, followed by Start My Server. This shall restart your Jupyter Hub container and refresh your tokens.
Please let me know if that resolves the issues.
Best,
Komal
-
This reply was modified 2 months ago by
Komal Thareja.
Looks like you are using Jupyter Hub. Could you please try File -> Hub Control Panel -> Stop My Server, followed by Start My Server. This shall restart your container and refresh your tokens.
Please let me know if that resolves the issues.
Best,
Komal
- WASH may have enough resources in aggregate, but individual hosts may lack the capacity needed for your requested VMs, resulting in the
-
AuthorPosts