[Komal Thareja]

@Mert / @Khawar,

I attempted to recover the VM last night and shut it down as part of the process. During the investigation, I noticed that the /home/ubuntu/.ssh directory was missing from the VM. I tried to restore the SSH keys to regain access, but subsequently found that the VM was no longer bootable and consistently failed with filesystem errors.

Further inspection showed that /etc/fstab on the VM had been modified:

LABEL=cloudimg-rootfs / ext4 discard,errors=remount-ro 0 1
LABEL=UEFI /boot/efi vfat umask=0077 0 1
vm0:/myvol /gss glusterfs defaults,_netdev,nofail 0 0

I attempted to revert the /etc/fstab changes, but was unable to recover to a bootable state. It appears these modifications may have been introduced as part of your experiment, possibly unintentionally.

Please be mindful when making system-level changes during experiments. In some cases, recovery is not possible if the VM state has been significantly altered and the changes are not fully known.

Best,

Komal

[Komal Thareja]

Hi Meshal,

Could you please share your notebook? I was able to successfully create a slice with 100 VMs distributed across 6–8 sites without any issues. If you can share your notebook, I’d be happy to try reproducing the error and work on resolving it.

Best,

Komal

[Komal Thareja]

Hi Nishanth,

FABRIC currently has only three IPv4-capable sites: TOKY, BRIST, and FIU. BlueField devices are not available at BRIST or TOKY. I’ll work on reproducing the issue and investigate the connectivity problem on the IPv6 sites, and I’ll share my findings once I have more information. Thanks for your patience!

Best,
Komal

[Komal Thareja]

Hi Fatih,

The PCI devices had been disconnected from your VMs, but I’ve now re-attached them. You should be able to see them on your VM.

I’ll review the logs to determine what caused this. In the meantime, if you’re able to share any operations or actions triggered as part of your experiment, that would be very helpful in narrowing down the issue. Thanks so much for your help!

Best,
Komal

[Komal Thareja]

Hi Suhib,

To use P4 Tofino switches, your project lead can request the Switch.P4 permission directly through the FABRIC portal.

FABRIC also offers BlueField-3 DPUs, which support P4, as well as FPGAs—both of these resources similarly require explicit permission requests. You can find details on project roles and permissions here:
https://learn.fabric-testbed.net/knowledge-base/fabric-user-roles-and-project-permissions/#project-permissions

You may also want to explore several example artifacts available at:
https://artifacts.fabric-testbed.net/artifacts/

Best,

Komal

[Komal Thareja]

Hi Tejas,

Are you still observing the SSH issues?

Best,

Komal

[Komal Thareja]

Hi Tejas,

Could you please check the logs: /tmp/fablib/fablib.log and also check if your bastion keys are not expired?

Please re-run jupyter-examples-*/configure_and_validate.ipynb notebook to renew your SSH keys. Please try creating the slice again after this.

Best,

Komal

 

[Komal Thareja]

hi Jianzhang,

Could you please share your slice details – slice id?

Best,

Komal

• This reply was modified 3 weeks, 5 days ago by Komal Thareja.

[Komal Thareja]

Maintenance is complete and testbed is operational.

Best regards,

The FABRIC Team

[Komal Thareja]

@yoursunny – Please consult this page for detailed CPU specifications.

FABRIC Host Configurations by Site

Please let us know if you need anything else.

Best,

Komal

[Komal Thareja]

Yes Fatih, you should be able to modify, extend this slice without issues.

Best,

Komal

[Komal Thareja]

Hi Jacob,

Thanks for reaching out. Based on the last discussions around this within our team, Intel RAPL metrics were not driven up into the VM plane, and we decided not to expose energy monitoring (or estimations) via the VM APIs. So at this time, this capability is not available on FABRIC.

Best regards,
Komal

[Komal Thareja]

Hi Fatih,

I hope you are doing well too, and thank you for reaching out with the detailed description.

The slice is currently in a StableError state because some slivers encountered failures during the earlier modification attempt and were subsequently closed. This behavior is intentional: FABRIC reports the slice as StableError to preserve visibility into past sliver failures, even after the problematic resources have been cleaned up.

At this point, since the affected network service has already been closed and no longer appears in the slice, there is no further action required to delete it. Your remaining active resources should continue to function normally, and their operation is not impacted by the slice being in StableError. In other words, this state is informational rather than blocking.

If you would like to proceed with a clean state, the recommended option is to create a new slice with the desired topology. Otherwise, you may continue using the current slice as-is if the active slivers meet your needs.

Please let us know if you have any additional questions or if you’d like help recreating the slice or network service.

Best regards,
Komal

[Komal Thareja]

Hi Ilya,

Happy New Year to you as well!

The slice is still up and running; however, your project has expired, which is preventing the CM from issuing tokens.

I’ve requested Michael to extend your project, and that should resolve the issue shortly.

Best regards,
Komal

[Komal Thareja]

Maintenance has been completed! Testbed is open to use!

Best,

Komal

[Author]

Posts