[Komal Thareja]

Hi Durbek,

Thank you for reporting this issue. This seems to be an issue in older version of Fablib and has been addressed in the later versions.

The default notebook creates VM with default_rocky_8 image (which comes with NetworkManager) and as part of post_boot_config Fablib tries to stop it. This is not the case for the default_ubuntu images. The failure observed is when Fablib is trying to stop the Network manager.

I would recommend following steps to help address this issue:

1. Please update fabric_config/requirements.txt in your JH container to include the following statement:
fabrictestbed-extensions==1.3.3

2. Restart your JH container

3. Try your experiment again, you may see some traces like which can be ignored.
sudo: nmcli: command not found

4. You would also need to update the cells where IP addresses are configured on the VMs for ubuntu to include following two steps after ip addr show:

Node1:

stdout, stderr = node1.execute('sudo apt install net-tools')
stdout, stderr = node1.execute(f'sudo ifconfig {node1_iface.get_os_interface()} up')

Node2:

stdout, stderr = node2.execute('sudo apt install net-tools')
stdout, stderr = node2.execute(f'sudo ifconfig {node2_iface.get_os_interface()} up')

Thanks,
Komal

[Komal Thareja]

Maintenance on FABRIC-Network AM – Completed

[Komal Thareja]

Hi George,

Could you please share the Slice ID for your slice? Please note that Slice Delete is an asynchronous operation. Slice might show stay in StableError for a bit until all the associated resources with it are deleted/released.

NOTE: Deletion was a synchronous operation earlier but was made asynchronous recently as part of performance optimizations.

Thanks,
Komal

• This reply was modified 2 years, 3 months ago by Komal Thareja.

[Komal Thareja]

User can not delete Dead and Closing slices. Dead and Closing slices do not have any Active resources associated with them as they have been deleted. Portal does not show Dead and Closing slices by default. If you do want to see the Dead and Closing slices on the portal you can check the check box “Include Dead and Closing” Slices. We periodically delete the Dead and Closing slices older than 30 days.

[Komal Thareja]

Thank you Charles for reporting this issue. It’s a bug in the code. We will fix it.

[Komal Thareja]

JupyterHub has been updated with the CLI containing the fixes for JSON outputs.

[Komal Thareja]

Thank you Timothy and Sajith for your contributions! Appreciate it! I have approved the PR and merged to the master branch. Will create a new release and push to Jupyter Hub.

[Komal Thareja]

Slices in Closing state do not have any resources allocated to them. They can be ignored. They stay in the Closing state as the confirmation for the delete response from the various Sites is not received (potentially because of the Kafka timeouts). These slices do not prevent the creation of new slices with the same name. We are working to improve the Kafka timeouts and would work to address the issue of Closing slices in the upcoming release.

Slices in Configuring state would prevent the creation of a new slice with the same name. I checked all the slices listed above and they are all in Closing state.

Thank you so much for sharing your observations and help us make the testbed better.
Please do share the slice id of the slices in Configuring state if you do run into them again.

[Komal Thareja]

No worries, I was able to reproduce this issue by creating a slice from the GUI. We will work on resolving this, thank you for helping us identify this problem and make the testbed better. Appreciate your feedback.

[Komal Thareja]

We are working to investigate this behavior and would appreciate if you please share the slice id of the slice created from the portal? This would help us gather the required information quickly.

Appreciate your help!

Thanks

[Komal Thareja]

Also, from previous conversation, I found that the slice was renewed on Sep 8. Could you please let us know when did you loose SSH connectivity?

[Komal Thareja]

Hello Gregory,

I can see both the VMs for your slice are ACTIVE. However, I am unable to SSH into them. I will seek help from operations team and keep you posted.

Reservation ID: da5faa94-1c50-41b7-abf5-47578a82b87b Slice ID: 4cb0a209-ca5e-4479-b0b7-e192fe257964
Resource Type: VM Notices: Reservation da5faa94-1c50-41b7-abf5-47578a82b87b (Slice MySliceSep12B(4cb0a209-ca5e-4479-b0b7-e192fe257964) Graph Id:d854653d-cf7b-407d-ae4d-149c5113262b Owner:aroy59@asu.edu) is in state (Active,None_)
Start: 2022-09-13 15:50:24 +0000 End: 2022-09-18 03:54:26 +0000 Requested End: 2022-09-18 03:54:26 +0000
Units: 1 State: Active Pending State: None_
Sliver: {'capacities': '{ core: 32 , ram: 128 G, disk: 100 G}', 'capacity_allocations': '{ core: 32 , ram: 128 G, disk: 100 G}', 'capacity_hints': '{ instance_type: fabric.c32.m128.d100}', 'image_ref': 'default_rocky_8', 'image_type': 'qcow2', 'label_allocations': '{ instance: instance-0000132a, instance_parent: tacc-w3.fabric-testbed.net}', 'management_ip': '129.114.110.85', 'name': 'CMBS4Node_tacc2', 'node_map': "('508c3fa3-df17-41ab-bb95-fdf71c105a61', '8QQBZC3')", 'reservation_info': '{"error_message": "", "reservation_id": "da5faa94-1c50-41b7-abf5-47578a82b87b", "reservation_state": "Active"}', 'site': 'TACC', 'type': 'VM'}
('CMBS4Node_tacc2-nic2', {'capacity_allocations': '{ unit: 1 }', 'details': 'Mellanox ConnectX-6 VPI MCX653 dual port 100Gbps', 'label_allocations': '{ bdf: 0000:e2:0f.6}', 'model': 'ConnectX-6', 'name': 'CMBS4Node_tacc2-nic2', 'node_map': "('508c3fa3-df17-41ab-bb95-fdf71c105a61', '8QQBZC3-slot7')", 'type': 'SharedNIC'})

Reservation ID: 2571ddf7-f838-46b8-9095-ed0d36cfec55 Slice ID: 4cb0a209-ca5e-4479-b0b7-e192fe257964
Resource Type: L2STS Notices: Reservation 2571ddf7-f838-46b8-9095-ed0d36cfec55 (Slice MySliceSep12B(4cb0a209-ca5e-4479-b0b7-e192fe257964) Graph Id:d854653d-cf7b-407d-ae4d-149c5113262b Owner:aroy59@asu.edu) is in state (Active,None_)
Start: 2022-09-13 15:50:25 +0000 End: 2022-09-18 03:54:26 +0000 Requested End: 2022-09-18 03:54:26 +0000
Units: 1 State: Active Pending State: None_
Sliver: {'layer': 'L2', 'name': 'net1', 'node_map': "('508c3fa3-df17-41ab-bb95-fdf71c105a61', 'node+tacc-data-sw:ip+192.168.16.3-ns')", 'reservation_info': '{"error_message": "", "reservation_id": "2571ddf7-f838-46b8-9095-ed0d36cfec55", "reservation_state": "Active"}', 'type': 'L2STS'}
{'capacities': '{ unit: 1 }', 'label_allocations': '{ mac: 02:FA:69:DF:46:DD, vlan: 2121, local_name: HundredGigE0/0/0/5, device_name: ncsa-data-sw}', 'labels': '{ mac: 02:FA:69:DF:46:DD, vlan: 2121, local_name: HundredGigE0/0/0/5, device_name: ncsa-data-sw}', 'name': 'CMBS4Node_ncsa1-CMBS4Node_ncsa1-nic1-p1', 'node_map': "('508c3fa3-df17-41ab-bb95-fdf71c105a61', 'port+ncsa-data-sw:HundredGigE0/0/0/5')", 'type': 'ServicePort'}
{'capacities': '{ unit: 1 }', 'label_allocations': '{ mac: 06:34:B9:B2:55:E2, vlan: 2124, local_name: HundredGigE0/0/0/9, device_name: tacc-data-sw}', 'labels': '{ mac: 06:34:B9:B2:55:E2, vlan: 2124, local_name: HundredGigE0/0/0/9, device_name: tacc-data-sw}', 'name': 'CMBS4Node_tacc2-CMBS4Node_tacc2-nic2-p1', 'node_map': "('508c3fa3-df17-41ab-bb95-fdf71c105a61', 'port+tacc-data-sw:HundredGigE0/0/0/9')", 'type': 'ServicePort'}

Reservation ID: 430e4832-f048-4368-b8b6-51ff6a5b6932 Slice ID: 4cb0a209-ca5e-4479-b0b7-e192fe257964
Resource Type: VM Notices: Reservation 430e4832-f048-4368-b8b6-51ff6a5b6932 (Slice MySliceSep12B(4cb0a209-ca5e-4479-b0b7-e192fe257964) Graph Id:d854653d-cf7b-407d-ae4d-149c5113262b Owner:aroy59@asu.edu) is in state (Active,None_)
Start: 2022-09-13 15:50:24 +0000 End: 2022-09-18 03:54:26 +0000 Requested End: 2022-09-18 03:54:26 +0000
Units: 1 State: Active Pending State: None_
Sliver: {'capacities': '{ core: 32 , ram: 128 G, disk: 100 G}', 'capacity_allocations': '{ core: 32 , ram: 128 G, disk: 100 G}', 'capacity_hints': '{ instance_type: fabric.c32.m128.d100}', 'image_ref': 'default_rocky_8', 'image_type': 'qcow2', 'label_allocations': '{ instance: instance-0000072a, instance_parent: ncsa-w1.fabric-testbed.net}', 'management_ip': '2620:0:c80:1001:f816:3eff:feef:a24c', 'name': 'CMBS4Node_ncsa1', 'node_map': "('508c3fa3-df17-41ab-bb95-fdf71c105a61', 'F1FSZB3')", 'reservation_info': '{"error_message": "", "reservation_id": "430e4832-f048-4368-b8b6-51ff6a5b6932", "reservation_state": "Active"}', 'site': 'NCSA', 'type': 'VM'}
('CMBS4Node_ncsa1-nic1', {'capacity_allocations': '{ unit: 1 }', 'details': 'Mellanox ConnectX-6 VPI MCX653 dual port 100Gbps', 'label_allocations': '{ bdf: 0000:a1:1f.2}', 'model': 'ConnectX-6', 'name': 'CMBS4Node_ncsa1-nic1', 'node_map': "('508c3fa3-df17-41ab-bb95-fdf71c105a61', 'F1FSZB3-slot6')", 'type': 'SharedNIC'})

• This reply was modified 2 years, 4 months ago by Komal Thareja.

[Komal Thareja]

Hello Xusheng,

This error is typically observed when either FABRIC_PROJECT_ID environment variable is not set or an explicit parameter project_id is not passed when creating instance of FablibMananger.

Could you please verify and try again and share the snapshot of the error observed?

Thanks,
Komal

[Komal Thareja]

Thank you for let us know! We will work on fixing the example. Please use the snippet below to renew the slice.
Hope this helps!

 

import datetime

#Set end host to now plus 1 day
now = datetime.now(timezone.utc)
end_date = now + timedelta(days=6).strftime("%Y-%m-%d %H:%M:%S %z")

try:
slice = fablib.get_slice(name=slice_name)

slice.renew(end_date)
except Exception as e:
print(f"Exception: {e}")

[Komal Thareja]

Could you please invoke the show_config() and share the output before invoking new slice?
Also, could you please verify that the token file has refresh_token and json looks like below:

P.S: You may already have it in correct format.

{"id_token": "YOUR ID Token",
"refresh_token": "YOUR Refresh Token",
"created_at": "timestamp"}

fablib.default_fablib_manager = FablibManager()

fablib.show_config()

show_config would list the token file being used, please verify that it is correct.

Thanks,

Komal

[Author]

Posts