[Komal Thareja]

Hi Sadat,

Could you please provide following information?

• Slice ID
• Verify and report the status of the Slice from Portal via Experiments -> My Slices
• Verify and report if any errors observed in /tmp/fablib/fablib.log from JH container.
• Verify Bastion SSH Keys are not expired
  • Check via portal Experiments -> Manage SSH Keys -> Bastion Key
  • In JH Container – Run jupyter-examples-rel1.8*/configure_and_validate.ipynb – this shall renew your expired keys
  • Please try your slice again if your Bastion Keys are expired.

Thanks,

Komal

• This reply was modified 2 months, 3 weeks ago by Komal Thareja.

[Komal Thareja]

Hi Samia,

I verified all the VMs in your slice are accessible via SSH. The error you are noticing is probably because of expired bastion keys. Could you please try to re-execute the notebook: jupyter-examples-rel1.8.1/configure_and_validate/configure_and_validate.ipynb ?

This shall renew your bastion keys. If you are doing SSH from your laptop, please download the renewed bastion keys from /home/fabrirc/work/fabric_config directory after executing the notebook above to replace the keys in .ssh directory.

Please let me know if you run into any issues or have questions.

Thanks,

Komal

[Komal Thareja]

Hello,

Could you please share your Slice ID?

Thanks,

Komal

[Komal Thareja]

FABRIC only allows SSH and few ICMP messages over the management interface. Hosting services on management network is not recommended. Instead, we recommend using data plane network for your service.

FABRIC serves as a secure sandbox, allowing students and researchers to experiment with potentially disruptive and vulnerable software architectures in a protected environment. When connecting external devices, such as laptops or servers, to nodes within a slice, it is crucial to employ secure methods like SSH tunnels. A Jupyter notebook example illustrates how to create SSH tunnels through the FABRIC bastion host. Alternatively, users can utilize personal VPNs like Tailscale for secure connections example notebook.

Exposing ports to the entire Internet is restricted, reserved only for exceptional cases where alternative solutions are not viable. Moreover, users undertaking such capabilities are responsible for deploying, maintaining, and ensuring the security of experiments, akin to a production data center. IPv4Ext and IPv6Ext services facilitate these capabilities.

For newcomers, getting acquainted with SSH tunnels is recommended due to their simplicity and security. If users have additional questions or require further guidance, they are encouraged to reach out.

Best,

Komal

[Komal Thareja]

Hi Mike,

Could you try specifying multiple VLANs when adding the Facility port and then access the interfaces as shown below?

facility_port = slice.add_facility_port(name=facility_port, site=facility_port_site, vlan=['3300', '3301'])  
facility_port_interface_1 = facility_port.get_interfaces()[0]  
facility_port_interface_2 = facility_port.get_interfaces()[1]  

 

P.S: Please change the vlans accordingly in the example above.

Thanks,
Komal

• This reply was modified 3 months, 2 weeks ago by Komal Thareja.

[Komal Thareja]

Hi Mike,

Could you please check following files and ensure they point to the new key and have the right bastion username?

/home/fabric/work/fabric_config/fabric_rc

/home/fabric/work/fabric_config/ssh_config

Also, just verified that the VM in question seems to have been deleted.

Thanks,

Komal

[Komal Thareja]

Hi Mike,

This could possibly be because your Bastion Key has expired. Could you please try running the  notebook jupyter-examples-rel1.8.1/configure_and_validate/configure_and_validate.ipynb again?

This should renew the keys. If error persists, could you please share if you see any errors in /tmp/fablib/fablib.log

Thanks,

Komal

• This reply was modified 3 months, 3 weeks ago by Komal Thareja.

[Komal Thareja]

Hi Tejas,

Could you please share your slice id?

 

Thanks,

Komal

[Komal Thareja]

Glad to hear it worked, Mike! The default container should function properly as well—I’ll give it a try.

The Bleeding Edge container includes the latest released software versions, so I’d recommend using that.

Beyond Bleeding Edge contains the newest software from the main branch, including ongoing development changes, which may introduce some bugs.

Thanks,

Komal

[Komal Thareja]

Hi Mike,

I just tried the example notebook you mentioned. It worked for me using the Beyond Bleeding Edge Container. Could you please share which JH container are you using? Also, if possible share the Slice ID where you see this error or which Facility Ports you ran into this error with?

 

Thanks,

Komal

[Komal Thareja]

Hi Sourya,

There was a power outage at SALT and is being recovered. We will let you know as soon as the recovery is complete.

Thanks,

Komal

[Komal Thareja]

Hi Tanay,

We are targeting a release for either Summer or Fall and will share more details once our plans are finalized.

Thanks,

Komal

[Komal Thareja]

Hi Sankalpa,

Based on our logs, this slice has been renewed three times in descending order of time. During the renewal attempt on 2025-02-15 13:46:54,320, Client3 (be97d870-3299-418e-ba17-a1ddcab06bdb) could not be renewed because a required component was likely allocated to another future slice. Since requesting slices in the future is enabled, this allocation prevented the renewal of that particular VM. However, other available resources were successfully extended.

The lease for Client3 ended on 2025-02-19 05:05:37 UTC, after which the VM was closed/deleted. The latest renewal request was issued on 2025-02-19 22:04:39, which was after the lease expiration, making the VM ineligible for renewal.

Logs:

• 2025-02-19 22:04:39,589 – CFEL Slice event: Renewal attempt by prj:b3cffedd-ddb4-43ee-b57d-459b768e14ca (usr: sankalpatimilsina12@gmail.com)
• 2025-02-19 22:09:37,543 – CFEL Slice event: Renewal attempt by prj:b3cffedd-ddb4-43ee-b57d-459b768e14ca (usr: sankalpatimilsina12@gmail.com)
• 2025-02-15 13:46:54,320 – CFEL Slice event: Renewal attempt by prj:b3cffedd-ddb4-43ee-b57d-459b768e14ca (usr: sankalpatimilsina12@gmail.com)

Reservation Details:

• Reservation ID: be97d870-3299-418e-ba17-a1ddcab06bdb
• Slice ID: 6acbc4aa-4b6e-44e3-b7c0-8c2f33de46c4
• Resource Type: VM
• Status: Closed (Last update: *Insufficient resources – Renew failed: Component of type ConnectX-6 with PCI Address 0000:a1:04.7 is already allocated to another reservation on node GDXYNF3).

Start: 2025-01-24 17:18:11 UTC
End: 2025-02-19 05:05:37 UTC
Requested End: 2025-02-28 13:46:54 UTC

Let me know if you need further clarification.

Best,

Komal

[Komal Thareja]

Hi Luca,

I reviewed your slice and noticed that during the last renewal, two of the VMs could not be renewed due to insufficient resources. As a result, they were not extended and have now transitioned to a Closed state, meaning they have been deleted. Below is a snapshot for reference.

Please note that since users can request slices for future use, it’s possible that renewing an active slice may fail if resources are already reserved for future allocations.

You can check the current state of your slice using the following code:

from fabrictestbed_extensions.fablib.fablib import FablibManager as fablib_manager
fablib = fablib_manager()
slice_name = "Slice INT slice - DALL + LOSA"
slice = fablib.get_slice(slice_name)
slice.list_nodes()

Additionally, if the renewal was triggered via JupyterHub (JH), this information has also been communicated to you there.


Reservation ID: b0ff1824-011d-4225-a748-371ddf6eb5e4 Slice ID: 831a0115-8e9e-4854-bbcb-d12022a878aa
Resource Type: VM Notices: Reservation b0ff1824-011d-4225-a748-371ddf6eb5e4 (Slice INT slice - DALL + LOSA(831a0115-8e9e-4854-bbcb-d12022a878aa) Graph Id:f6a2c692-b430-4a01-95d6-2f2343320dea Owner:s317694@studenti.polito.it) is in state (Closed,None_) (Last ticket update: Insufficient resources : ['ram', 'disk'])


Reservation ID: dc4456ff-f768-4915-8c7e-97696b2fcc21 Slice ID: 831a0115-8e9e-4854-bbcb-d12022a878aa
Resource Type: VM Notices: Reservation dc4456ff-f768-4915-8c7e-97696b2fcc21 (Slice INT slice - DALL + LOSA(831a0115-8e9e-4854-bbcb-d12022a878aa) Graph Id:f6a2c692-b430-4a01-95d6-2f2343320dea Owner:s317694@studenti.polito.it) is in state (Closed,None_) (Last ticket update: Insufficient resources : ['ram', 'disk'])


Thanks,

Komal

1 user thanked author for this post.

Luca Cetino

[Komal Thareja]

Hi,

I was able to run the notebook .

Could you please share your Slice ID?

Additionally, could you post your inquiries in the FABRIC General Questions and Discussion forum?

Thanks,

Komal

[Author]

Posts