[Komal Thareja]

@Manas – It looks like you are requesting: Cores=20, RAM=128GB, Disk=2000GB. Even though we may have an overall storage available to account for such a request, some of your VMs are causing the Worker to be exhausted. Specifically Disk requested cannot be served.

I was able to create this slice with Disk usage set to 500GB instead on SALT but your storage volume doesn’t exist there. Could you please try this slice with lower disk?

[Komal Thareja]

Duplicate: https://learn.fabric-testbed.net/forums/topic/unable-to-allocate-resources-after-the-updates-maintenance/

[Komal Thareja]

Hello,

I was debugging this and noticed that VMs requested by your slice are being mapped to capacities indicated below which seem to exhaust the worker where your request lands. The capacity mapping from requested to allocated seems strange.
I am unable to reproduce this. Could you please share your notebook?

Also, could you please share the output of command: pip3 list | grep fabric from your environment?
We expect the following versions to be present.

fabric                        3.0.0
fabric-credmgr-client         1.3.2
fabric-fim                    1.4.2
fabric-fss-utils              1.4.0
fabric-orchestrator-client    1.4.3
fabrictestbed                 1.4.3
fabrictestbed-extensions      1.4.0

Capacity Allocations for your slice: cluster_gatk(d8bfce5c-c721-4d41-a7fa-e8d658a40a43)

'capacities': '{ core: 2 , ram: 8 G, disk: 10 G}',                      ===> Requested by User
'capacity_allocations': '{ core: 16 , ram: 128 G, disk: 500 G}',        ===> Allocated by Orchestrator
'capacity_hints': '{ instance_type: fabric.c16.m128.d500}'

Thanks,
Komal

• This reply was modified 2 years, 2 months ago by Komal Thareja.

[Komal Thareja]

Could you please verify from the Portal, that project for which storage volumes were created has the Tag: Component.Storage available? Also, Please make sure you are using the same project ID in JH.

If your project does not have the tag: Component.Storage, Please raise a request to enable that for your project.

[Komal Thareja]

Looks like your project doesn’t have permissions for the Persistent Storage.
You would need to request Component.Storage permissions and persistent storage from the Portal by going to the Contact Us tab.

Thanks
Komal

[Komal Thareja]

Hi,

Could you please try to use the following notebook as indicated on jupyter-examples-rel1.4.1/start_here.ipynb ?

Persistent Storage: Connect to your project’s persisent storage volume.

Thanks,
Komal

[Komal Thareja]

Closing the topic

[Komal Thareja]

Maintenance is complete. UTAH and MICH sites are also available for use.

[Komal Thareja]

Hello,

Could you please run the configure notebook from ‘Jupiter-examples-rel1.4.1’ directory and then try the hello world example again?

From the screenshot it looks like environment was rolled back to Release 1.3.8.

Thanks,
Komal

[Komal Thareja]

Update is complete. All sites except MICH and UTAH are available for use. We will share another update when MICH and UTAH are available.

NOTE: Please install/upgrade to latest version of fabrictestbed-extensions if you are accessing FABRIC resources from outside the JupyterHub environment.

pip install fabrictestbed-extensions==1.4.0

[Komal Thareja]

Update is complete!

[Komal Thareja]

This looks like an issue with the Project ID setting in fabric_config/fabric_rc. Could you please re-run the notebook: Configure Environment and verify that Project ID being set is same as the Project Id in the portal?

Setup Environment
Configure Environment: Configure you Environment including creating the fabric_rc and ssh_config files.

• This reply was modified 2 years, 3 months ago by Komal Thareja.

[Komal Thareja]

Hello Nagmat,

Please Restart your Jupyter Container via File -> Hub Control Panel -> Stop My server, followed by Start My Server.
Please let us know if you still face this error.

Thanks,
Komal

[Komal Thareja]

This maintenance is now complete.

[Komal Thareja]

Hi Durbek,

Thank you for reporting this issue. This seems to be an issue in older version of Fablib and has been addressed in the later versions.

The default notebook creates VM with default_rocky_8 image (which comes with NetworkManager) and as part of post_boot_config Fablib tries to stop it. This is not the case for the default_ubuntu images. The failure observed is when Fablib is trying to stop the Network manager.

I would recommend following steps to help address this issue:

1. Please update fabric_config/requirements.txt in your JH container to include the following statement:
fabrictestbed-extensions==1.3.3

2. Restart your JH container

3. Try your experiment again, you may see some traces like which can be ignored.
sudo: nmcli: command not found

4. You would also need to update the cells where IP addresses are configured on the VMs for ubuntu to include following two steps after ip addr show:

Node1:

stdout, stderr = node1.execute('sudo apt install net-tools')
stdout, stderr = node1.execute(f'sudo ifconfig {node1_iface.get_os_interface()} up')

Node2:

stdout, stderr = node2.execute('sudo apt install net-tools')
stdout, stderr = node2.execute(f'sudo ifconfig {node2_iface.get_os_interface()} up')

Thanks,
Komal

[Author]

Posts