1. Komal Thareja

Komal Thareja

Forum Replies Created

Viewing 15 posts - 1 through 15 (of 586 total)
  • Author
    Posts
  • Komal Thareja
    Moderator

      Hi,

      Thank you for the detailed report and timeline. It helped us find the problem quickly.

      I apologize for the inconvenience.

      What happened

      – 10/31 renew: A bug in how lease end times were compared made your network services fail to extend, even though the VMs they’re attached to extended fine. This is fixed, and should not impact any future extend requests.
      – 11/03 outage: Because of that failed renew, the network AM’s copy of your slice still had the old 10/31 expiry. An automated cleanup job then treated the slice as expired and removed it from the network AM’s records, even though your re-created services were still active. From that point the network AM couldn’t manage network services for this slice. That’s why all six links stopped working, and why every attempt to delete or re-add them failed with Slice Not Found. Nothing you did caused this. We’re also fixing the cleanup job so it can’t remove a slice that still has active services.

      Your VMs are not affected. All 19 nodes are still running. The network interfaces belong to the VMs, not to the network services, so your in-VM IP addresses and FRR configuration will carry over once the networks are re-created.

      We have resynced the network AM for your slice and clearing the stale VLAN configuration on the RUTG, PRIN, WASH and STAR switches.

      Please use the code below to re-add network services. It deletes any leftover network services, then re-creates net1-4 and the six router links on the same interfaces and subnets as before, and runs a quick ping check.

      # readd_networks.py — re-create net1-4 and the 6 router links in OSPF_Routing_Param4
      from ipaddress import IPv4Network
      from fabrictestbed_extensions.fablib.fablib import FablibManager

      SLICE_NAME = “OSPF_Routing_Param4”

      # name -> (type, [node interface names], subnet or None)
      NETWORKS = {
      “net1”: (“L2Bridge”, [“router1-nic_local-p1”, “rutg1-nic_local-p1”, “rutg2-nic_local-p1”,
      “rutg3-nic_local-p1”, “rutg4-nic_local-p1”, “rutg5-nic_local-p1”], None),
      “net2”: (“L2Bridge”, [“router2-nic_local-p1”, “prin1-nic_local-p1”, “prin2-nic_local-p1”,
      “prin3-nic_local-p1”], None),
      “net3”: (“L2Bridge”, [“router3-nic_local-p1”, “wash1-nic_local-p1”, “wash2-nic_local-p1”,
      “wash3-nic_local-p1”, “wash4-nic_local-p1”], None),
      “net4”: (“L2Bridge”, [“router4-nic_local-p1”, “star1-nic_local-p1”, “star2-nic_local-p1”,
      “star3-nic_local-p1”], None),
      “link_r1_r2”: (“L2STS”, [“router1-r1-r2-p1”, “router2-r2-r1-p1”], “192.168.101.0/24”),
      “link_r1_r3”: (“L2STS”, [“router1-r1-r3-p1”, “router3-r3-r1-p1”], “192.168.102.0/24”),
      “link_r1_r4”: (“L2STS”, [“router1-r1-r4-p1”, “router4-r4-r1-p1”], “192.168.103.0/24”),
      “link_r2_r3”: (“L2STS”, [“router2-r2-r3-p1”, “router3-r3-r2-p1”], “192.168.104.0/24”),
      “link_r2_r4”: (“L2STS”, [“router2-r2-r4-p1”, “router4-r4-r2-p1”], “192.168.105.0/24”),
      “link_r3_r4”: (“L2STS”, [“router3-r3-r4-p1”, “router4-r4-r3-p1”], “192.168.106.0/24”),
      }

      fablib = FablibManager()

      # ———- Step 1: remove any leftover network services ———-
      slice = fablib.get_slice(name=SLICE_NAME)
      print(“State before:”, slice.get_state())
      existing = slice.get_network_services()
      print(“Deleting:”, [n.get_name() for n in existing])
      for net in existing:
      net.delete()

      if existing:
      # If this times out, do NOT re-run submit(). Check
      # fablib.get_slice(name=SLICE_NAME).get_state() and wait for StableOK/ModifyOK.
      slice.submit()

      slice = fablib.get_slice(name=SLICE_NAME)
      left = [n.get_name() for n in slice.get_network_services()]
      assert not left, f”Network services still present, please stop and contact us: {left}”
      print(“All network services removed; state:”, slice.get_state())

      # ———- Step 2: re-create the networks on the same interfaces ———-
      ifaces = {i.get_name(): i for n in slice.get_nodes() for i in n.get_interfaces()}

      for name, (ns_type, if_names, subnet) in NETWORKS.items():
      missing = [x for x in if_names if x not in ifaces]
      if missing:
      raise RuntimeError(f”{name}: interfaces not found {missing}; available: {sorted(ifaces)}”)
      slice.add_l2network(name=name, type=ns_type,
      interfaces=[ifaces[x] for x in if_names],
      subnet=IPv4Network(subnet) if subnet else None)

      slice.submit(wait=True, progress=True)

      # ———- Step 3: verify ———-
      slice = fablib.get_slice(name=SLICE_NAME)
      print(“State after:”, slice.get_state())
      for net in slice.get_network_services():
      print(f”{net.get_name():12s} {net.get_type()} {net.get_reservation_state()} {net.get_error_message()}”)

      # Router-to-router reachability over each link (adjust if your .1/.2 assignment differs)
      PEERS = {“router1”: [“192.168.101.2”, “192.168.102.2”, “192.168.103.2”],
      “router2”: [“192.168.104.2”, “192.168.105.2”],
      “router3”: [“192.168.106.2″]}
      for node_name, ips in PEERS.items():
      node = slice.get_node(name=node_name)
      for ip in ips:
      out, _ = node.execute(f”ping -c 3 -W 2 {ip} | tail -2″, quiet=True)
      print(f”{node_name} -> {ip}: {out.strip()}”)

      A few notes:

      – net1-4 are re-created too. Their current copies have the same problem as the links, so re-creating everything at once is the cleanest fix.
      – Timeouts: if submit() times out, don’t run it again. Earlier, the retries after timeouts created duplicate requests: your two attempts became four rounds of submissions. Check the slice state and wait for it to settle instead.
      – Peer IPs: the ping check assumes .1 is the lower-numbered router on each link. Adjust PEERS if yours are the other way round.
      – OSPF: if adjacencies don’t come back on their own within a minute or two, run vtysh -c “clear ip ospf process” in each router container.

      If anything in the script fails, send us the output and we’ll take it from there. Sorry for the disruption to your experiment.

      Best regards,
      Komal

      Komal Thareja
      Moderator

        Hi Haribabu,

        Could you please try creating your slice again?

        Best,

        Komal

        Komal Thareja
        Moderator

          An unexpected infrastructure outage yesterday caused these issues. More details are available in the outage announcement.

          Testbed services have been restored, and your slices should now be visible.

          Best,
          Komal

          in reply to: Testbed Outage #10113
          Komal Thareja
          Moderator

            Dear Experimenters,

            Control Framework services have been restored, and you can now resume slice operations.

            We apologize for the inconvenience.

            Best,
            Komal

            in reply to: Slices stuck in closing; internal server error #10100
            Komal Thareja
            Moderator

              Looking at it now. May i please ask which JH container option are you running?

              I mean which option did you select when you logged in to JH?

              Best,

              Komal

               

              in reply to: Slices stuck in closing; internal server error #10096
              Komal Thareja
              Moderator
                1. WASH may have enough resources in aggregate, but individual hosts may lack the capacity needed for your requested VMs, resulting in the Insufficient Resources error.
                2. Resources associated with a Closing slice are released, as they are for a Dead slice. Delayed responses to resource closure requests can sometimes leave a slice stuck in the Closing state.
                3. This may have been a transient error. The slice looks healthy, and all requested resources are now Active.

                Best,

                Komal

                in reply to: Slices stuck in closing; internal server error #10097
                Komal Thareja
                Moderator
                  1. WASH may have enough resources in aggregate, but individual hosts may lack the capacity needed for your requested VMs, resulting in the Insufficient Resources error.
                  2. Resources associated with a Closing slice are released, as they are for a Dead slice. Delayed responses to resource closure requests can sometimes leave a slice stuck in the Closing state.
                  3. This may have been a transient error. The slice looks healthy, and all requested resources are now Active.

                  Best,

                  Komal

                  in reply to: FIU Tofino switch refused (Insufficient resources: [unit]) #10092
                  Komal Thareja
                  Moderator

                    Hi,

                    1. Your switch request was rejected because the resource was unavailable, as indicated by the error Insufficient resources: [unit]. We currently have only five Tofino switches, and demand is high. Please check the Resource Calendar in LoomAI for future availability. Because FABRIC is a distributed system, another request may have acquired the switch between the availability check and the processing of your request. I have verified that another slice holds the switch and was created at the similar time when your slice requests were triggered.
                    2. All resources associated with slices in the Closing or Dead state are released.

                    Best,
                    Komal

                    in reply to: Network service name[ ..] must be unique.] #10091
                    Komal Thareja
                    Moderator

                      Hi Vakhob,

                      If you run into a token error, please restart your JH container via

                      File -> Hub Control Panel -> Stop My Server followed by Start My Server.

                      Also, to avoid the Unique name error, please restart the Notebook kernel.

                      If not using already, please consider using the new recommended way of using FABRIC via loomai.net.

                      Best,

                      Komal

                      in reply to: Link publications to a FABRIC project #10081
                      Komal Thareja
                      Moderator

                        Hi Romaric,

                        We may not have the on Portal yet. But if you share which publication needs to be associated with the corresponding project we can update that for now. We will also work to address this on the portal.

                        Best,

                        Komal

                        in reply to: NVIDIA BlueField-3 DPUs #10077
                        Komal Thareja
                        Moderator

                          Hi Ivan,

                          Could you please share your slice ID?

                          Best,

                          Komal

                          in reply to: Bluefield Configuration Issue #10055
                          Komal Thareja
                          Moderator

                            Hi Stephen,

                            There was a mismatch in the code for configure function and the actual BFB image on the host. I pushed a fix for this and updated the notebook too. The notebook should work now.

                            Thank you for sharing your observation and helping us address the issue. Appreciate it!

                            Best,

                            Komal

                            in reply to: token error – while reserving a slice #10054
                            Komal Thareja
                            Moderator

                              I’m glad to hear the issue has been resolved! Thank you for sharing the error message. We’ll work on making it clearer and more helpful.

                              Best,
                              Komal

                              in reply to: token error – while reserving a slice #10051
                              Komal Thareja
                              Moderator

                                Hi Garegin,

                                Could you please share the error they are encountering?

                                I verified that the configure_and_validate.ipynb notebook works without any issues with the latest API.

                                We also recommend using loomai.net as the new interface for accessing FABRIC. LoomAI is an AI-assisted graphical interface that also provides access to the existing JupyterHub environment.

                                Also, please try File -> Hub Control Panel -> Stop My Server, followed by Start My Server. This shall restart your Jupyter Hub container and refresh your tokens.

                                Please let me know if that resolves the issues.

                                Best,

                                Komal

                                in reply to: Unable to request slice initiation. #10044
                                Komal Thareja
                                Moderator

                                  Looks like you are using Jupyter Hub. Could you please try File -> Hub Control Panel -> Stop My Server, followed by Start My Server. This shall restart your container and refresh your tokens.

                                  Please let me know if that resolves the issues.

                                  Best,

                                  Komal

                                Viewing 15 posts - 1 through 15 (of 586 total)