[Komal Thareja]

Hi Manas,

Glad to hear the issue is resolved. Please find my responses inline below.

1. I have recently changed the bastion key, so I don’t get why the key expired.

[KT] Could you please share how this was done? Please note if the bastion keys are re-created via portal, user is responsible to upload them to JH.

1. I am able to ssh into the node, but the bastion key I have in my laptop (local) is different from the bastion key in the Jupyter hub.

[KT] This should not be an issue. Users can have multiple bastion keys.

1. Is multiple bastion keys possible?

[KT] Yes, this not an issue.

1. How to check the expiration date of the bastion key in my Jupyter hub, I know how to check the expiration date of the bastion key that I use to ssh into the node from my local machine.

[KT] You can check the bastion key expiry on the portal or run the validate_and_configure.ipynb to validate/update your config as needed.

Hope this helps!

Thanks,

Komal

[Komal Thareja]

Hi Manas,

This typically happens when either your bastion keys are expired or sliver keys used by fablib don’t match the keys inside the slivers.

Could you please run this notebook: jupyter-examples-rel1.6.1/configure_and_validate.ipynb to ensure your config is valid and if bastion keys are expired they are regenerated?

Please try list/show on your slice after that and let us know if you still face the problem.

P.S: Verified that all your VMs are accessible via SSH.

Thanks,

Komal

• This reply was modified 1 year, 10 months ago by Komal Thareja.

[Komal Thareja]

Hello Zheyi,

The VMs from your slices are being provisioned on eduky-w12.fabric-testbed.net This host is heavily used and is reporting Page Faults and Memory issues. We are investigating that and have placed eduky-w12.fabric-testbed.net in Maintenance.

Please try creating a slice on EDUKY and let us know if you still see the issue.

Thanks,

Komal

1 user thanked author for this post.

Zheyi Qin

[Komal Thareja]

Good morning Violet and Jackson,

We have deployed a fix for the OVS Bridges. Experiments with OVS Bridges can now be conducted using NIC_Basic with the following host considerations.

 

Host Considerations:

Because of constraints imposed by NVIDIA/Mellanox, when utilizing NIC_Basic for an OVS bridge experiment, it is advisable to deploy the VM responsible for running the bridge on a separate host from the VMs linked to the bridge.

Additionally, it’s worth noting that this condition does not apply to NIC_ConnectX_5 and NIC_ConnectX_6 configurations.

 

Example Notebook:

Updated example is available at: https://github.com/fabric-testbed/jupyter-examples/blob/main/fabric_examples/complex_recipes/openvswitch/openvswitch.ipynb

Thanks,

Komal

[Komal Thareja]

Hi Vaiden,

VM is up and accessible via SSH. Could you please check if your bastion keys are expired?

Thanks,

Komal

[Komal Thareja]

Hi Kriti,

By default, fablib marks the dataplane interfaces as unmanaged by NetworkManager. So NetworkManager is not logging the interface down operation. This was done deliberately as having an interface managed by NetworkManager kept overriding the IP address configuration applied by fablib. Please note this behavior is specific to rocky images which enables NetworkManager by default. Ubuntu doesn’t use NetworkManager by default.

We disable NetworkManager on rocky to be consistent with Ubuntu and leave it to the user to change this behavior as they see fit.

 

Thanks,

Komal

[Komal Thareja]

Hello Yifang,

Could you please share the output of the following commands? This seems to be an issue local to your environment.

cat /home/fabric/work/fabric_config/fabric_rc and ls -ltr /home/fabric/work/fabric_config

Also, please try to Restart your JH container via File -> Hub Control Panel -> Stop My Container followed by Start My Container.

Thanks,

Komal

[Komal Thareja]

Maintenance is complete and the network model has been updated.

[Komal Thareja]

Hello Khawar,

Could you please check if you have any entries for fabrictestbed-extensions in /home/fabric/work/fabric_config/requirements.txt ? If so, please remove those entries and restart your container via File -> Hub Control Panel -> Stop My container followed by Start My Container.

This should clear the issue. Please let us know if the problem persists.

Thanks,

Komal

[Komal Thareja]

Could you please remove the file /home/fabric/work/fabric_config/ssh_config and re-run the jupyter-examples-rel1.6.1/configure_and_validate.ipynb notebook? This shall fix the config issue.

Thanks,
Komal

[Komal Thareja]

Could you please share the output of the command cat /home/fabric/work/fabric_config/ssh_config ?

If this file already existed, the notebook wouldn’t have made any changes to it. Looks like line 5 has some issues.

Sharing example contents of this file below with BASTION USERNAME MASKED:

UserKnownHostsFile /dev/null
StrictHostKeyChecking no
ServerAliveInterval 120
Host bastion.fabric-testbed.net
User YOUR_BASTION_USER_NAME
ForwardAgent yes
Hostname %h
IdentityFile /home/fabric/work/fabric_config/fabric_bastion_key
IdentitiesOnly yes
Host * !bastion.fabric-testbed.net
ProxyJump YOUR_BASTION_USER_NAME@bastion.fabric-testbed.net:22

• This reply was modified 1 year, 11 months ago by Komal Thareja.

[Komal Thareja]

The error observed is because of the incomplete config. You can run the configure notebook from the path jupyter-examples-rel1.6.1/configure_and_validate.ipynb without impacting your experiments.

This only sets up your environment. Experiments from rel-1.5.4 would still work after the config.

Thanks,

Komal

1 user thanked author for this post.

Robin Schelstraete

[Komal Thareja]

Hi Robin,

Config notebook has been updated to avoid such errors.

Could you please run the notebook: jupyter-examples-rel1.6.1/configure_and_validate.ipynb ?

This should update your bastion keys if expired or do not exist. Please try creating a slice and SSH access again after that from console as well as the notebook. Please restart the notebook kernel after the keys are updated.

Thanks,

Komal

[Komal Thareja]

Attaching the screenshot for Chameleon Lease

[Komal Thareja]

Please create a Lease to reserve a host on Chameleon via Project -> Reservations -> Leases -> Create Lease.

Once the lease is created, click on the lease, you will Reservation section on it, Copy the Id from there.

This is the Id you need to use in the notebook. Hope this helps.

If you create the Server on Chameleon manually. Please set the IP address and the routes on the server as below:

ip addr add 10.130.162.2/24 dev eth1

Add route: route add -net 10.130.162.0/24 dev eth1

Change the IP and interface as per your FabNet subnet.

[Author]

Posts