[Komal Thareja]

Hi Natty,

Apologies for the inconvenience! I looked at your Slice and can confirm from the logs the deletion was triggered by user either via Fablib/Portal.

2024-07-10 21:30:15,216 - CFEL Slice event slc:54466474-bcc2-45ef-a86a-4696b7d1bc5e create by prj:073ee843-2310-45bd-a01f-a15d808827dc usr:4d5326ac-b002-444f-bb44-3b406a038be5:nm3833@nyu.edu:49c50f84eab437103fa7f7863bfbbd5bbe4e303ea017b2912ce0df418b61a3df compute vms:5,cores:48,p4s:0; sites GATECH; components SmartNIC:1,SharedNIC:4; services L2Bridge:0,L2Bridge:0; vmdetails C32/R256/D25:1,C4/R32/D10:4

2024-07-11 14:25:52,610 - CFEL Slice event slc:54466474-bcc2-45ef-a86a-4696b7d1bc5e delete by prj:073ee843-2310-45bd-a01f-a15d808827dc usr:4d5326ac-b002-444f-bb44-3b406a038be5:nm3833@nyu.edu:49c50f84eab437103fa7f7863bfbbd5bbe4e303ea017b2912ce0df418b61a3df

This possibly may have happened because of the Bug reported here With the maintenance next week the delete bug would be addressed.

Thanks,

Komal

• This reply was modified 1 year, 2 months ago by Komal Thareja.

[Komal Thareja]

Hi Nishant,

It appears that a network service has leaked. In a distributed system like our testbed, encountering some leaked resources is not unusual. We plan to deploy updates in the coming week to address this issue. In the meantime, I recommend introducing a delay between deletion and recreation, as the resources are distributed across the testbed.

For now, I have cleaned up the leaked services, so provisioning should work.

Also, if possible could you please share your notebook or code- snippet that might help reproduce this state. Would be super helpful to debug and address this issue? Appreciate your help with this!

Best regards,

Komal

[Komal Thareja]

We are deploying 1.7 next week which would contain a fix for this issue. Apologies for the inconvenince!

Thanks,

Komal

[Komal Thareja]

Hi Nishant,

VM requested on GATECH identified by ID: 3d425fc6-0f44-4e98-a0cc-d9ee9358cb8f cannot be allocated. Looks like you are requesting CX6 there which are only available on GATECH-w3 which is currently under maintenance. Hence, CF is unable to find any nodes to serve this reservation.

Hope this helps!

Thanks,

Komal

[Komal Thareja]

Hi Nirmala,

FABLIB API supports 3 modes of configuration for the VMs.
– Manual: Manual configuration does not require any additional steps before the slice request is submitted.

– Auto: Automatic configuration requires specify a subnet for the network and setting the interface’s mode to auto using the iface1.set_mode(‘auto’) function before submitting the request. With automatic configuration, FABlib will allocate an IP from the network’s subnet and configure the device during the post boot configuration stage. Optionally, you can add routes to the node before submitting the request.

– User Defined (config): User defined configuration requires specifying a subnet for the network and specifying the IP to use for each interface before the request is submitted. You can signal FABlib to configure the user defined IPs by setting the interface’s mode to config using the iface1.set_mode(‘config’) function before submitting the request. With user defined configuration, FABlib will use the IP defined by the user and configure the device during the post boot configuration stage. Optionally, you can add routes to the node before submitting the request.

Examples for each mode of configuration are available via Start Here:

Reboot Persistence:

Assuming the manual configuration is done via ip addr commands, it is not reboot persistent and the onus lies on the user to save/apply the config again post reboot.

For modes auto and config, FABLIB maintains the IP address information in the meta data for each of VMs maintained in UserData JSON object saved in the Fabric Information Model for the VM. In both these modes, the configuration can be fetched and re-applied using the code block:

slice = fablib.get_slice(slice_name)
for n in slice.get_nodes()
n.config()

Hope this helps! Please let me know if you have any feedback.

Thanks,
Komal

• This reply was modified 1 year, 2 months ago by Komal Thareja.
• This reply was modified 1 year, 2 months ago by Komal Thareja.

[Komal Thareja]

Hi Vaiden,

Not sure how your slice was setup. If you interfaces were configured in auto mode. You should be able to do the following to reapply the config.


slice = fablib.get_slice(slice_name)
for n in slice.get_nodes():
n.config()


 

Thanks,

Komal

• This reply was modified 1 year, 2 months ago by Komal Thareja.

[Komal Thareja]

Hi Vaiden,

I have checked the logs and don’t see any errors. Looks like your slice was modified multiple times.

Some of the resources closed on 2024-05-18 18:27:07 +0000 while all other had expiry date set to 2024-11-02 19:45:33 +0000. Is it possible that the slice deletion was triggered by you on accident?

I have no evidence of software deleting it due to expiry as per the logs.

Could you please share if there were any recent actions taken on the slice? Trying to see if this can be recreated.

Appreciate your help with this.

Thanks,

Komal

[Komal Thareja]

Hi Vaiden,

Unfortunately, it is not possible to recover the slivers once deleted. I will look at why the slice was closed before the end date. However, I do suspect that Extend/Renew may have failed for certain slivers. I will look more and share details here.

Thanks,

Komal

[Komal Thareja]

Good morning Fraida,

I’ve pushed a fix to address this issue. Could you please try again using the Beyond Bleeding Edge container on JH and let us know if the problem persists? Your help is greatly appreciated!

Thanks,

Komal

[Komal Thareja]

Hi Pilar,

Could you please check if your bastion keys are expired via on the Portal ->Experiments -> Manage SSH Keys -> Bastion Keys? If so, Please re-run the notebook jupyter-examples-rel1.6.1/configure_and_validate.ipynb

This should renew your bastion keys. Please try creating your slice again after this.

Thanks,

Komal

[Komal Thareja]

Hi Nishant,

Installing fablib from main branch should work. It is using the fabrictestbed==1.5.9 dependency which has the fix.

fabrictestbed==1.5.9 is built from llt branch. I will work on merging this branch to main as well.

Thanks,

Koma

[Komal Thareja]

Hi Fraida,

Yes, FABRIC now supports Slices using OVS Bridges using NIC_Basic. An example notebook can be found here: https://github.com/fabric-testbed/jupyter-examples/blob/main/fabric_examples/complex_recipes/openvswitch/openvswitch.ipynb

We do have following constraint though for this to work.

Host Considerations:

Because of constraints imposed by NVIDIA/Mellanox, when utilizing NIC_Basic for an OVS bridge experiment, it is advisable to deploy the VM responsible for running the bridge on a separate host from the VMs linked to the bridge.

Additionally, it’s worth noting that this condition does not apply to NIC_ConnectX_5 and NIC_ConnectX_6 configurations.

Thanks,

Komal

[Komal Thareja]

Thank you for sharing this Sunjay, we will fix this in the next version.

Thanks,

Komal

[Komal Thareja]

Posting an update to close the loop.

Had a quick Zoom meeting with Laura to resolve this. The issue seemed to be bastion keys, removing contents of fabric_config and re-running configure_and_validate.ipynb resolved the issue.

@Laura – Please let us know if you run into any issues!

Thanks,

Komal

[Komal Thareja]

Hi Shoaib,

Which network are you trying to setup – layer2 or layer3? Please share your slice id to help us investigate this further.

In addition, please take a look at the examples available from start_here.ipynb

All the networking examples have three configurations:

• Auto – FABLIB automatically configures IPs and routes
• Manual – User explicitly configures IPs and routes
• Config – User explicitly specifies the IPs/subnets to choose and FABLIB automatically configures IPs and routes

Please let us know if you still have questions or concerns.

Thanks,

Komal

• This reply was modified 1 year, 3 months ago by Komal Thareja.
• This reply was modified 1 year, 3 months ago by Komal Thareja.

[Author]

Posts