[Komal Thareja]

Hi Sree,

VMs cannot communicate with each other over the private IPs assigned to interfaces connected to the management network. The interfaces with addresses in the 10.* range belong to this management network. Inter-VM communication should instead occur over the data plane network, which in your case is the L2Bridge network.

I reviewed your slice and noticed that you have three VMs and two L2Bridge networks configured. However, the IP addresses on the VM interfaces are not set up correctly. Each network must use a different subnet, and the corresponding VM interfaces should be assigned IP addresses from those respective subnets.

Please refer to the following example notebook, which demonstrates how to correctly configure the network:
jupyter-examples-*/fabric_examples/fablib_api/create_l2network_basic/create_l2network_basic_auto.ipynb

Make sure to use separate subnets for each network and assign the appropriate IPs to the VM interfaces so that communication works properly.

Best,
Komal

[Komal Thareja]

Hi Sree,

Could you please share your slice ID so we can look at it? In addition, please check some of the following examples available via jupyter-examples-*/start_here.ipynb that may be useful.

 

Thanks,

Komal

• This reply was modified 4 days, 12 hours ago by Komal Thareja.
• This reply was modified 4 days, 12 hours ago by Komal Thareja.

[Komal Thareja]

Hi Fatih,

Could you please try changing the following files:

/home/fabric/work/fabric_config/ssh_config

/home/fabric/work/fabric_config/fabric_config

change bastion.fabric-testbed.net to bastion-ncsa-1.fabric-testbed.net in both the files.

Reload the kernel of your notebook and try the node.execute

Thanks,

Komal

[Komal Thareja]

Hi Fatih,

Could you please check if you see any errors in the /tmp/fablib/fablib.log?

Also, could you please share which Jupyter Container are you using?

Best,

Komal

[Komal Thareja]

@Mert / @Khawar,

I attempted to recover the VM last night and shut it down as part of the process. During the investigation, I noticed that the /home/ubuntu/.ssh directory was missing from the VM. I tried to restore the SSH keys to regain access, but subsequently found that the VM was no longer bootable and consistently failed with filesystem errors.

Further inspection showed that /etc/fstab on the VM had been modified:

LABEL=cloudimg-rootfs / ext4 discard,errors=remount-ro 0 1
LABEL=UEFI /boot/efi vfat umask=0077 0 1
vm0:/myvol /gss glusterfs defaults,_netdev,nofail 0 0

I attempted to revert the /etc/fstab changes, but was unable to recover to a bootable state. It appears these modifications may have been introduced as part of your experiment, possibly unintentionally.

Please be mindful when making system-level changes during experiments. In some cases, recovery is not possible if the VM state has been significantly altered and the changes are not fully known.

Best,

Komal

[Komal Thareja]

Hi Meshal,

Could you please share your notebook? I was able to successfully create a slice with 100 VMs distributed across 6–8 sites without any issues. If you can share your notebook, I’d be happy to try reproducing the error and work on resolving it.

Best,

Komal

[Komal Thareja]

Hi Nishanth,

FABRIC currently has only three IPv4-capable sites: TOKY, BRIST, and FIU. BlueField devices are not available at BRIST or TOKY. I’ll work on reproducing the issue and investigate the connectivity problem on the IPv6 sites, and I’ll share my findings once I have more information. Thanks for your patience!

Best,
Komal

[Komal Thareja]

Hi Fatih,

The PCI devices had been disconnected from your VMs, but I’ve now re-attached them. You should be able to see them on your VM.

I’ll review the logs to determine what caused this. In the meantime, if you’re able to share any operations or actions triggered as part of your experiment, that would be very helpful in narrowing down the issue. Thanks so much for your help!

Best,
Komal

[Komal Thareja]

Hi Suhib,

To use P4 Tofino switches, your project lead can request the Switch.P4 permission directly through the FABRIC portal.

FABRIC also offers BlueField-3 DPUs, which support P4, as well as FPGAs—both of these resources similarly require explicit permission requests. You can find details on project roles and permissions here:
https://learn.fabric-testbed.net/knowledge-base/fabric-user-roles-and-project-permissions/#project-permissions

You may also want to explore several example artifacts available at:
https://artifacts.fabric-testbed.net/artifacts/

Best,

Komal

[Komal Thareja]

Hi Tejas,

Are you still observing the SSH issues?

Best,

Komal

[Komal Thareja]

Hi Tejas,

Could you please check the logs: /tmp/fablib/fablib.log and also check if your bastion keys are not expired?

Please re-run jupyter-examples-*/configure_and_validate.ipynb notebook to renew your SSH keys. Please try creating the slice again after this.

Best,

Komal

 

[Komal Thareja]

hi Jianzhang,

Could you please share your slice details – slice id?

Best,

Komal

• This reply was modified 1 month, 2 weeks ago by Komal Thareja.

[Komal Thareja]

Maintenance is complete and testbed is operational.

Best regards,

The FABRIC Team

[Komal Thareja]

@yoursunny – Please consult this page for detailed CPU specifications.

FABRIC Host Configurations by Site

Please let us know if you need anything else.

Best,

Komal

[Komal Thareja]

Yes Fatih, you should be able to modify, extend this slice without issues.

Best,

Komal

[Author]

Posts