[Komal Thareja]

Thank you @yoursunny for sharing the details on how to request more disk space on experiment VMs.

The /home/fabric/work directory (1GB) in the JupyterHub environment serves as persistent storage for code, notebooks, scripts, and other materials related to configuring and running experiments, including the addition of extra Python modules. However, it is not designed to handle large datasets or output files.

Please consider removing un-needed files to avoid this error.

Additionally, if you need more disk space in the Jupyter Hub Container, I recommend setting up your own FABRIC environment on your laptop or machine to run your experiments. This approach will allow you to capture more data and reduce reliance on Jupyter Hub.

Consider one of the following options:

• Running JupyterHub Container locally as described here.
• configuring a local Python environment for the FABRIC API as described here, and run the notebooks locally.

Best,

Komal

• This reply was modified 9 months, 2 weeks ago by Komal Thareja.

[Komal Thareja]

Hi Abdulhadi,

Thank you for sharing the details. I was able to reproduce it and have posted a fix. If you try the Beyond Bleeding Edge JH container, this should work. I plan to update the other JH containers with the fix sometime next week.

Thanks,

Komal

[Komal Thareja]

It appears there may be an issue with list_nodes, which could be affecting the display.

However, I suspect the P4 switch you requested at SITE is currently unavailable. You can verify this by checking your slice in the Portal. If your slice is in a Dead or Closed state, enable the “Include Dead/Closing Slices” option.

When you click on your slice, you may see an error message such as “insufficient resources.” 

Please choose a different site or look at the P4 availability via following code or check from Portal-> Resources Overview screen.

p4_column_name = 'p4-switch_available'

[site2] = fablib.get_random_sites(count=1, filter_function=lambda x: x[p4_column_name] > 0)

Thanks,

Komal

[Komal Thareja]

Hi Ajay,

Your slice is currently in a Dead state, and all associated resources have been released.

If you encounter this issue again, please share details of an active slice or VM.

Thanks,

Komal

[Komal Thareja]

Hi,

Could you please share the Slice ID for your slice?

 

Thanks,

Komal

1 user thanked author for this post.

Abdulhadi Sahin

[Komal Thareja]

Hi Arash,

Could you please try deleting the key from the portal and run fablib.verify_and_configure()?

Thanks,

Komal

1 user thanked author for this post.

Arash

[Komal Thareja]

Peter’s fix should still work Nirmala!

Thanks,

Komal

[Komal Thareja]

Hi Garegin,

fabrictestbed-extensions==1.9.1 is pushed to pypi and available in Bleeding Edge JH container.

This version contains the fix for the get_interface.

Thanks,

Komal

[Komal Thareja]

Hi Nirmala/Peter,

fabrictestbed-extensions==1.9.1 is pushed to pypi and available in Bleeding Edge JH container.

This version contains the fix for the get_interfaces.

Thanks,

Komal

[Komal Thareja]

Hi Nirmala,

Could you please try using the fablib from this branch: https://github.com/fabric-testbed/fabrictestbed-extensions/tree/rel1.9.1 ?

nw.get_interfaces() should work now.

I plan to push this main branch Monday. I’ll keep you posted.

Please let me know if this resolves the issue.

Thanks,

Komal

1 user thanked author for this post.

Peter Willis

[Komal Thareja]

Hi Philip,

Thanks for pointing that out — noted! I’ll discuss internally to see if we can support specifying bandwidth in Mbps via the API.

In the meantime, you could consider using tools like tc to shape traffic at a more granular level to meet your 30 Mbps requirement.

Best regards,
Komal

[Komal Thareja]

Hi Ajay,

node.os_reboot() is recommended to be executed only if you are doing CPU pinning or NUMA tuning. This failed because your VM was already in shutoff state. If the intent is to just reboot the VM, please use sudo reboot via node.execute(). Also, what kind of workload is your application/experiment running? We are noticing some kernel level CPU locks on the host where your VM is running. We want to investigate if something from your experiment is triggering this. Could you please share more details about the experiment workload being executed on this VM?

Appreciate your help with this!

Thanks,

Komal

[Komal Thareja]

Hi Ajay,

Your VM was in a shutoff state, which I’ve now restored. Could you please share the notebook that outlines the type of workload you’re running on this VM? We’ve observed similar instances with your slices in the past, so having this information would help us identify the root cause of your VMs shutting down.

Thanks,
Komal

[Komal Thareja]

Hi Nirmala,

Could you please change the permissions of the key as indicated in the error message using the command below via terminal in your JH container? This shall fix the issue.

chmod 600 /home/fabric/work/fabric_config/Nirmala

Thanks,

Komal

[Komal Thareja]

Hi Garegin,

Thank you for sharing your observation. There was performance fix to improve how the interfaces are handled in the fablib. I suspect this may have introduced this issue. I will investigate this and post a fix once ready.

For now, please use the workaround as you suggested. Apologies for the inconvenience.

Thanks,

Komal

[Author]

Posts