[Komal Thareja]

Hi Fengping,

Your second slice failed with the error: Insufficient resources as depicted below. Please note that slice deletion is not synchronous, it may take some time for all the resources associated with a slice to be deleted. Please consider adding slight delay between subsequent slice creation attempts if both the slices are requesting resources from the same site which might not have been released yet by the first slice.

Resource Type: VM Notices: Reservation 113cd41c-26df-461e-8dc9-f93ed92fcebf (Slice ServiceXSlice(66a78e70-ecf2-41e7-be12-740561904991) Graph Id:cc871ebc-e290-4b44-ab36-046d3cd2da00 Owner:fengping@uchicago.edu) is in state (Closed,None_) (Last ticket update: Insufficient resources : ['disk'])

 

For the second slice, you can view the failure reasons from the portal, by select the check box ‘Include Dead/Closed Slices`.

Please try creating the slice again and let us know if you still see errors.

 

Thanks,

Komal

• This reply was modified 2 years ago by Komal Thareja.

[Komal Thareja]

Sorry, for not indicating this before, you can restart the container from File -> Hub Control Panel -> Stop My Server, then logout and login.

[Komal Thareja]

Could you please remove the entries for fabrictestbed-extensions from ~/work/fabric_config/requirements.txt and restart your container? Please try your notebooks after the restart.

Thanks,

Komal

[Komal Thareja]

Hello Acheme,

Could you please share the output of the following commands? Also, which container are you using?

pip list | grep fabric

cat ~/work/fabric_config/requirements.txt

Thanks,

Komal

[Komal Thareja]

Hello Thushari,

We are looking into this and have found that in a race condition when list_resources times out, you may observe the output as shared by you. We are debugging this and would work on a fix. In the meanwhile, could you please explicitly pass a site name when adding a node to your slice and try it?

Also, please try both 1.4.6 and 1.5.1 containers and let us know if it works for any of the options.

Thanks,

Komal

[Komal Thareja]

It looks like an older version of fablib is running in your container. Could you please ensure that there are no entries for fabrictestbed-extensions in fabric_config/requirements.txt ? Please restart your JH container after that and try the notebook again.

Versions should something like below for 1.4.6 and 1.5.1 container options.

For 1.4.6 container:

pip list | grep fabric
fabric 3.1.0
fabric-credmgr-client 1.3.2
fabric-fim 1.4.14
fabric-fss-utils 1.4.0
fabric-orchestrator-client 1.4.7
fabrictestbed 1.4.7
fabrictestbed-extensions 1.4.6


For 1.5.1 container:


$ pip list | grep fabric
fabric 3.1.0
fabric-credmgr-client 1.5.0
fabric_fim 1.5.2
fabric_fss_utils 1.5.0
fabric-orchestrator-client 1.5.1
fabrictestbed 1.5.1
fabrictestbed-extensions 1.5.1


 

Thanks,

Komal

[Komal Thareja]

Hello Nagmat,

Could you please share the output of the following command from a terminal window in JH container?

pip list | grep fabric

 

Thanks,

Komal

• This reply was modified 2 years ago by Komal Thareja.

[Komal Thareja]

Hello,

Could you please ensure that configure_environment.ipynb notebook has been executed and your environment is setup? Also, please remove any entries for fabrictestbed-extensions you may have in fabric_config/requirements.txt and restart your container and attempt creating the slice again?

Thanks,

Komal

[Komal Thareja]

Credential Manager has been updated and the maintenance is complete!

[Komal Thareja]

SSL certs updated and maintenance is complete.

[Komal Thareja]

Thank you for reporting this! It’s a bug, for now I have patched the JH with the fix. I will push it to the github repo as well.

Thanks,

Komal

[Komal Thareja]

Closing the topic!

[Komal Thareja]

Maintenance is complete. Jupyter Hub has been updated to provide following different container options:

• FABRIC Examples v1.4.3, FABlib v1.4.1: released: 04/23/2023, stable: Spring 2023
• (default) FABRIC Examples v1.4.6, FABlib v1.4.4: released: 05/21/2023, stable: Summer 2023
• (bleeding edge) FABRIC Examples latest, FABlib latest, Release 1.4

Thanks,

Komal

[Komal Thareja]

Hi Nagmat,

Could you please check if your bastion keys are still valid? Also, could you please share and check for errors in /tmp/fabric/fablib.log ?

Thanks,
Komal

[Komal Thareja]

Hi Nagmat,

I don’t see any failures for this slice either. It seems like this slice was deleted on 2023-05-16 17:31:42 UTC.

However, I do see that, another slice created by you yesterday failed with “Insufficient resources error” which would explain the notebook behavior.

Reservation ID: fb7939b5-8f8b-4517-a975-3da648a190c3 Slice ID: 55c64491-4edc-44cc-885f-264f9fa4cbc2
Resource Type: VM Notices: Reservation fb7939b5-8f8b-4517-a975-3da648a190c3 (Slice basic_nagm01(55c64491-4edc-44cc-885f-264f9fa4cbc2) Graph Id:348882a5-575c-438e-a3aa-18c1c933794e Owner:nagmat@nevada.unr.edu) is in state (Closed,None_)  (Last ticket update: Insufficient resources : Component of type: ConnectX-5 not available in graph node: 8QTDZC3)
Start: 2023-05-22 19:28:49 +0000 End: 2023-05-23 19:28:47 +0000 Requested End: 2023-05-23 19:28:47 +0000

Thanks,
Komal

[Author]

Posts