Forum Replies Created
-
AuthorPosts
-
August 25, 2025 at 8:09 pm in reply to: CPU Pinning and Numa Tuning ConnectX_5/ConnectX_6 notebook not working #8848
Hi Rasman,
The dataplane was down at MASS. The link has been restored, could you please try running the iPerf3 cells again from the notebook. Apologies for the inconvenience.
Thanks,
Komal
August 25, 2025 at 11:42 am in reply to: CPU Pinning and Numa Tuning ConnectX_5/ConnectX_6 notebook not working #8845Hi Rasman,
I was able to run iperf3 optimized notebook without issues. I am unable to access your notebook. It says Page Not Found.
Could you please share your slice ID?
Thanks,
Komal
August 22, 2025 at 4:44 pm in reply to: JupyterLab Save Error: No Space Left on Device in FABRIC Slice #8841Thank you @yoursunny for sharing the details on how to request more disk space on experiment VMs.
The /home/fabric/work directory (1GB) in the JupyterHub environment serves as persistent storage for code, notebooks, scripts, and other materials related to configuring and running experiments, including the addition of extra Python modules. However, it is not designed to handle large datasets or output files.
Please consider removing un-needed files to avoid this error.
Additionally, if you need more disk space in the Jupyter Hub Container, I recommend setting up your own FABRIC environment on your laptop or machine to run your experiments. This approach will allow you to capture more data and reduce reliance on Jupyter Hub.
Consider one of the following options:
- Running JupyterHub Container locally as described here.
- configuring a local Python environment for the FABRIC API as described here, and run the notebooks locally.
Best,
Komal
-
This reply was modified 5 days, 10 hours ago by
Komal Thareja.
August 15, 2025 at 5:11 pm in reply to: Layer 2 Network Topology with Edgecore Wedge100BF-32X P4 Tofino Switch on FABRIC #8828Hi Abdulhadi,
Thank you for sharing the details. I was able to reproduce it and have posted a fix. If you try the Beyond Bleeding Edge JH container, this should work. I plan to update the other JH containers with the fix sometime next week.
Thanks,
Komal
August 15, 2025 at 2:56 pm in reply to: Layer 2 Network Topology with Edgecore Wedge100BF-32X P4 Tofino Switch on FABRIC #8826It appears there may be an issue with
list_nodes
, which could be affecting the display.However, I suspect the P4 switch you requested at SITE is currently unavailable. You can verify this by checking your slice in the Portal. If your slice is in a Dead or Closed state, enable the “Include Dead/Closing Slices” option.
When you click on your slice, you may see an error message such as “insufficient resources.”
Please choose a different site or look at the P4 availability via following code or check from Portal-> Resources Overview screen.
p4_column_name = 'p4-switch_available'
[site2] = fablib.get_random_sites(count=1, filter_function=lambda x: x[p4_column_name] > 0)
Thanks,Komal
August 15, 2025 at 2:47 pm in reply to: NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. #8825Hi Ajay,
Your slice is currently in a Dead state, and all associated resources have been released.
If you encounter this issue again, please share details of an active slice or VM.
Thanks,
Komal
August 15, 2025 at 2:43 pm in reply to: Layer 2 Network Topology with Edgecore Wedge100BF-32X P4 Tofino Switch on FABRIC #8824Hi,
Could you please share the Slice ID for your slice?
Thanks,
Komal
1 user thanked author for this post.
Hi Arash,
Could you please try deleting the key from the portal and run
fablib.verify_and_configure()
?Thanks,
Komal
1 user thanked author for this post.
August 11, 2025 at 1:45 pm in reply to: Cannot iterate over interfaces on a network in FABlib v1.9.0 #8795Peter’s fix should still work Nirmala!
Thanks,
Komal
Hi Garegin,
fabrictestbed-extensions==1.9.1
is pushed to pypi and available inBleeding Edge
JH container.This version contains the fix for the
get_interface
.Thanks,
Komal
August 11, 2025 at 1:34 pm in reply to: Cannot iterate over interfaces on a network in FABlib v1.9.0 #8792Hi Nirmala/Peter,
fabrictestbed-extensions==1.9.1
is pushed to pypi and available inBleeding Edge
JH container.This version contains the fix for the
get_interfaces
.Thanks,
Komal
August 8, 2025 at 1:16 pm in reply to: Cannot iterate over interfaces on a network in FABlib v1.9.0 #8789Hi Nirmala,
Could you please try using the fablib from this branch: https://github.com/fabric-testbed/fabrictestbed-extensions/tree/rel1.9.1 ?
nw.get_interfaces() should work now.
I plan to push this main branch Monday. I’ll keep you posted.
Please let me know if this resolves the issue.
Thanks,
Komal
1 user thanked author for this post.
August 6, 2025 at 10:33 pm in reply to: Guaranteed Capacity and Traffic Prioritization across the Sites #8782Hi Philip,
Thanks for pointing that out — noted! I’ll discuss internally to see if we can support specifying bandwidth in Mbps via the API.
In the meantime, you could consider using tools like
tc
to shape traffic at a more granular level to meet your 30 Mbps requirement.Best regards,
KomalAugust 5, 2025 at 2:27 pm in reply to: channel 0: open failed: connect failed: No route to host #8759Hi Ajay,
node.os_reboot()
is recommended to be executed only if you are doing CPU pinning or NUMA tuning. This failed because your VM was already in shutoff state. If the intent is to just reboot the VM, please usesudo reboot
vianode.execute()
. Also, what kind of workload is your application/experiment running? We are noticing some kernel level CPU locks on the host where your VM is running. We want to investigate if something from your experiment is triggering this. Could you please share more details about the experiment workload being executed on this VM?Appreciate your help with this!
Thanks,
Komal
August 5, 2025 at 11:32 am in reply to: channel 0: open failed: connect failed: No route to host #8757Hi Ajay,
Your VM was in a shutoff state, which I’ve now restored. Could you please share the notebook that outlines the type of workload you’re running on this VM? We’ve observed similar instances with your slices in the past, so having this information would help us identify the root cause of your VMs shutting down.
Thanks,
Komal -
AuthorPosts