[Komal Thareja]

Hello Thushari,

We are looking into this and have found that in a race condition when list_resources times out, you may observe the output as shared by you. We are debugging this and would work on a fix. In the meanwhile, could you please explicitly pass a site name when adding a node to your slice and try it?

Also, please try both 1.4.6 and 1.5.1 containers and let us know if it works for any of the options.

Thanks,

Komal

[Komal Thareja]

It looks like an older version of fablib is running in your container. Could you please ensure that there are no entries for fabrictestbed-extensions in fabric_config/requirements.txt ? Please restart your JH container after that and try the notebook again.

Versions should something like below for 1.4.6 and 1.5.1 container options.

For 1.4.6 container:

pip list | grep fabric
fabric 3.1.0
fabric-credmgr-client 1.3.2
fabric-fim 1.4.14
fabric-fss-utils 1.4.0
fabric-orchestrator-client 1.4.7
fabrictestbed 1.4.7
fabrictestbed-extensions 1.4.6


For 1.5.1 container:


$ pip list | grep fabric
fabric 3.1.0
fabric-credmgr-client 1.5.0
fabric_fim 1.5.2
fabric_fss_utils 1.5.0
fabric-orchestrator-client 1.5.1
fabrictestbed 1.5.1
fabrictestbed-extensions 1.5.1


 

Thanks,

Komal

[Komal Thareja]

Hello Nagmat,

Could you please share the output of the following command from a terminal window in JH container?

pip list | grep fabric

 

Thanks,

Komal

• This reply was modified 1 year, 9 months ago by Komal Thareja.

[Komal Thareja]

Hello,

Could you please ensure that configure_environment.ipynb notebook has been executed and your environment is setup? Also, please remove any entries for fabrictestbed-extensions you may have in fabric_config/requirements.txt and restart your container and attempt creating the slice again?

Thanks,

Komal

[Komal Thareja]

Credential Manager has been updated and the maintenance is complete!

[Komal Thareja]

SSL certs updated and maintenance is complete.

[Komal Thareja]

Thank you for reporting this! It’s a bug, for now I have patched the JH with the fix. I will push it to the github repo as well.

Thanks,

Komal

[Komal Thareja]

Closing the topic!

[Komal Thareja]

Maintenance is complete. Jupyter Hub has been updated to provide following different container options:

• FABRIC Examples v1.4.3, FABlib v1.4.1: released: 04/23/2023, stable: Spring 2023
• (default) FABRIC Examples v1.4.6, FABlib v1.4.4: released: 05/21/2023, stable: Summer 2023
• (bleeding edge) FABRIC Examples latest, FABlib latest, Release 1.4

Thanks,

Komal

[Komal Thareja]

Hi Nagmat,

Could you please check if your bastion keys are still valid? Also, could you please share and check for errors in /tmp/fabric/fablib.log ?

Thanks,
Komal

[Komal Thareja]

Hi Nagmat,

I don’t see any failures for this slice either. It seems like this slice was deleted on 2023-05-16 17:31:42 UTC.

However, I do see that, another slice created by you yesterday failed with “Insufficient resources error” which would explain the notebook behavior.

Reservation ID: fb7939b5-8f8b-4517-a975-3da648a190c3 Slice ID: 55c64491-4edc-44cc-885f-264f9fa4cbc2
Resource Type: VM Notices: Reservation fb7939b5-8f8b-4517-a975-3da648a190c3 (Slice basic_nagm01(55c64491-4edc-44cc-885f-264f9fa4cbc2) Graph Id:348882a5-575c-438e-a3aa-18c1c933794e Owner:nagmat@nevada.unr.edu) is in state (Closed,None_)  (Last ticket update: Insufficient resources : Component of type: ConnectX-5 not available in graph node: 8QTDZC3)
Start: 2023-05-22 19:28:49 +0000 End: 2023-05-23 19:28:47 +0000 Requested End: 2023-05-23 19:28:47 +0000

Thanks,
Komal

[Komal Thareja]

Hi Nagmat,

Could you please share slice id for the slice(s) if possible? Also, you could check in Portal under Experiment->Slices, please select “Include Dead/Closing” slices. Any errors that occurred in those slices should be displayed when you view the slice on the portal.

Thanks,
Komal

[Komal Thareja]

Maintenance was completed!

[Komal Thareja]

The maintenance has been completed.

[Komal Thareja]

Apologies I missed the screenshot, I see that you are trying to upload to the node without doing a get on it.

Could you please try adding the following statements before upload to ensure that node object has all the information from the slice?

s1 = slice.get_node(name="s1")  
s3 = slice.get_node(name="s3")

The above statements should be added after the cell which does a get_slice().

Also regarding the failed VM: h2 – unfortunately it is again because of disk un-availability. s2 and h2 are both requested on MICH and each VM is requesting 500G. Based on current allocation of MICH, only one can be provisioned.

This is because of a known issue: availability discrepancy between software and infrastructure. We are working on the plans to resolve this, until then if possible request VMs with smaller disks.

Thanks,
Komal

[Author]

Posts