[Komal Thareja]

I looked at your slices and found that you have 2 Dead Slices and 6 Closing Slices. All the slices are requesting VMs on a single site CERN. All the Slice requests are requesting either 120 or 60 cores. Regardless of the disk size, for core/ram requested these are mapped to the following flavors. Considering that there are other slices also on CERN site, your slice cannot be accommodated by single CERN site. Please consider either spanning your slice across multiple sites or reducing the size of the VM not only w.r.t disk but also cores/ram.

We currently only have a limited number of flavors and your core/ram request is being mapped to a huge disk.

core: 120 , ram: 480 G, ==>  fabric.c64.m384.d4000

core: 60 , ram: 360 G,  ==> fabric.c60.m384.d2000

NOTE: No manual cleanup is needed the software is behaving as designed.

Thanks,

Komal

[Komal Thareja]

Hi Fengping,

Your second slice failed with the error: Insufficient resources as depicted below. Please note that slice deletion is not synchronous, it may take some time for all the resources associated with a slice to be deleted. Please consider adding slight delay between subsequent slice creation attempts if both the slices are requesting resources from the same site which might not have been released yet by the first slice.

Resource Type: VM Notices: Reservation 113cd41c-26df-461e-8dc9-f93ed92fcebf (Slice ServiceXSlice(66a78e70-ecf2-41e7-be12-740561904991) Graph Id:cc871ebc-e290-4b44-ab36-046d3cd2da00 Owner:fengping@uchicago.edu) is in state (Closed,None_) (Last ticket update: Insufficient resources : ['disk'])

 

For the second slice, you can view the failure reasons from the portal, by select the check box ‘Include Dead/Closed Slices`.

Please try creating the slice again and let us know if you still see errors.

 

Thanks,

Komal

• This reply was modified 1 year, 7 months ago by Komal Thareja.

[Komal Thareja]

Sorry, for not indicating this before, you can restart the container from File -> Hub Control Panel -> Stop My Server, then logout and login.

[Komal Thareja]

Could you please remove the entries for fabrictestbed-extensions from ~/work/fabric_config/requirements.txt and restart your container? Please try your notebooks after the restart.

Thanks,

Komal

[Komal Thareja]

Hello Acheme,

Could you please share the output of the following commands? Also, which container are you using?

pip list | grep fabric

cat ~/work/fabric_config/requirements.txt

Thanks,

Komal

[Komal Thareja]

Hello Thushari,

We are looking into this and have found that in a race condition when list_resources times out, you may observe the output as shared by you. We are debugging this and would work on a fix. In the meanwhile, could you please explicitly pass a site name when adding a node to your slice and try it?

Also, please try both 1.4.6 and 1.5.1 containers and let us know if it works for any of the options.

Thanks,

Komal

[Komal Thareja]

It looks like an older version of fablib is running in your container. Could you please ensure that there are no entries for fabrictestbed-extensions in fabric_config/requirements.txt ? Please restart your JH container after that and try the notebook again.

Versions should something like below for 1.4.6 and 1.5.1 container options.

For 1.4.6 container:

pip list | grep fabric
fabric 3.1.0
fabric-credmgr-client 1.3.2
fabric-fim 1.4.14
fabric-fss-utils 1.4.0
fabric-orchestrator-client 1.4.7
fabrictestbed 1.4.7
fabrictestbed-extensions 1.4.6


For 1.5.1 container:


$ pip list | grep fabric
fabric 3.1.0
fabric-credmgr-client 1.5.0
fabric_fim 1.5.2
fabric_fss_utils 1.5.0
fabric-orchestrator-client 1.5.1
fabrictestbed 1.5.1
fabrictestbed-extensions 1.5.1


 

Thanks,

Komal

[Komal Thareja]

Hello Nagmat,

Could you please share the output of the following command from a terminal window in JH container?

pip list | grep fabric

 

Thanks,

Komal

• This reply was modified 1 year, 7 months ago by Komal Thareja.

[Komal Thareja]

Hello,

Could you please ensure that configure_environment.ipynb notebook has been executed and your environment is setup? Also, please remove any entries for fabrictestbed-extensions you may have in fabric_config/requirements.txt and restart your container and attempt creating the slice again?

Thanks,

Komal

[Komal Thareja]

Credential Manager has been updated and the maintenance is complete!

[Komal Thareja]

SSL certs updated and maintenance is complete.

[Komal Thareja]

Thank you for reporting this! It’s a bug, for now I have patched the JH with the fix. I will push it to the github repo as well.

Thanks,

Komal

[Komal Thareja]

Closing the topic!

[Komal Thareja]

Maintenance is complete. Jupyter Hub has been updated to provide following different container options:

• FABRIC Examples v1.4.3, FABlib v1.4.1: released: 04/23/2023, stable: Spring 2023
• (default) FABRIC Examples v1.4.6, FABlib v1.4.4: released: 05/21/2023, stable: Summer 2023
• (bleeding edge) FABRIC Examples latest, FABlib latest, Release 1.4

Thanks,

Komal

[Komal Thareja]

Hi Nagmat,

Could you please check if your bastion keys are still valid? Also, could you please share and check for errors in /tmp/fabric/fablib.log ?

Thanks,
Komal

[Author]

Posts