[Komal Thareja]

Could you please share the output of the command cat /home/fabric/work/fabric_config/ssh_config ?

If this file already existed, the notebook wouldn’t have made any changes to it. Looks like line 5 has some issues.

Sharing example contents of this file below with BASTION USERNAME MASKED:

UserKnownHostsFile /dev/null
StrictHostKeyChecking no
ServerAliveInterval 120
Host bastion.fabric-testbed.net
User YOUR_BASTION_USER_NAME
ForwardAgent yes
Hostname %h
IdentityFile /home/fabric/work/fabric_config/fabric_bastion_key
IdentitiesOnly yes
Host * !bastion.fabric-testbed.net
ProxyJump YOUR_BASTION_USER_NAME@bastion.fabric-testbed.net:22

• This reply was modified 8 months, 3 weeks ago by Komal Thareja.

[Komal Thareja]

The error observed is because of the incomplete config. You can run the configure notebook from the path jupyter-examples-rel1.6.1/configure_and_validate.ipynb without impacting your experiments.

This only sets up your environment. Experiments from rel-1.5.4 would still work after the config.

Thanks,

Komal

1 user thanked author for this post.

Robin Schelstraete

[Komal Thareja]

Hi Robin,

Config notebook has been updated to avoid such errors.

Could you please run the notebook: jupyter-examples-rel1.6.1/configure_and_validate.ipynb ?

This should update your bastion keys if expired or do not exist. Please try creating a slice and SSH access again after that from console as well as the notebook. Please restart the notebook kernel after the keys are updated.

Thanks,

Komal

[Komal Thareja]

Attaching the screenshot for Chameleon Lease

[Komal Thareja]

Please create a Lease to reserve a host on Chameleon via Project -> Reservations -> Leases -> Create Lease.

Once the lease is created, click on the lease, you will Reservation section on it, Copy the Id from there.

This is the Id you need to use in the notebook. Hope this helps.

If you create the Server on Chameleon manually. Please set the IP address and the routes on the server as below:

ip addr add 10.130.162.2/24 dev eth1

Add route: route add -net 10.130.162.0/24 dev eth1

Change the IP and interface as per your FabNet subnet.

[Komal Thareja]

The fix for BasicNICs is planned to be made available in the next software release 1.7 planned to be available by end of Spring semester.

Thanks,

Komal

• This reply was modified 8 months, 4 weeks ago by Komal Thareja.

[Komal Thareja]

I just ran this notebook: https://github.com/fabric-testbed/jupyter-examples/blob/main/fabric_examples/complex_recipes/Chameleon_Facility_Port/Chameleon_Facility_Port_fabnetv4.ipynb

No additional steps needed.

Thanks,

Komal

[Komal Thareja]

MASS is working as well. We checked your FABRIC nodes, Fabnet services seems to be connected properly and we can ping the gateway. FABRIC VMs in your slice can ping each other too.

Not sure how your Chameleon Server is setup.

You should see routes and interface setup something similar to below on your Chameleon Node:

cc@kthare10-fabric-stitch-server-1:~$ ip route list
default via 10.130.163.2 dev eno1np0 proto dhcp src 10.130.163.10 metric 100
10.128.0.0/10 via 10.130.163.1 dev eno1np0 proto dhcp src 10.130.163.10 metric 100
10.130.163.0/24 dev eno1np0 proto kernel scope link src 10.130.163.10
169.254.169.254 via 10.130.163.3 dev eno1np0 proto dhcp src 10.130.163.10 metric 100
cc@kthare10-fabric-stitch-server-1:~$
cc@kthare10-fabric-stitch-server-1:~$
cc@kthare10-fabric-stitch-server-1:~$
cc@kthare10-fabric-stitch-server-1:~$ ifconfig
eno1np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.130.163.10 netmask 255.255.255.0 broadcast 10.130.163.255
inet6 fe80::be97:e1ff:fec4:8e0 prefixlen 64 scopeid 0x20<link>
ether bc:97:e1:c4:08:e0 txqueuelen 1000 (Ethernet)
RX packets 4937 bytes 1058216 (1.0 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 4804 bytes 410390 (410.3 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

P.S: I did execute the cell indicated as “(Optionally) Add a Router and Attach it to the Subnet”.

 

• This reply was modified 8 months, 4 weeks ago by Komal Thareja.
• This reply was modified 8 months, 4 weeks ago by Komal Thareja.
• This reply was modified 8 months, 4 weeks ago by Komal Thareja.

[Komal Thareja]

Hello Violet,

Thank you for informing us about this. The OpenVswitch notebook has been uploaded to the Jupyter Examples and is now accessible on GitHub here.

Please be aware that, due to limitations imposed by NVIDIA/Mellanox on how bridging is managed, the VM running the OVS switch must utilize dedicated NICs instead of BasicNICs. This example assumes that condition. We have identified an alternative solution currently undergoing testing. We will keep you updated once a solution is deployed to make this work with BasicNICs.

Appreciate your understanding.

Best regards,

Komal

[Komal Thareja]

Hi Sanjana,

I am able to reproduce this issue on MASS. But I was able to get this to work on other sites like SEAT, PSC. Could you please use a different site like SEAT or PSC while we investigate this issue. I will keep you updated with the findings for MASS.

Thank you for sharing your observations and helping us make the testbed better.

Thanks,

Komal

[Komal Thareja]

@Khawar – Could you please try accessing JH again and let me know how it goes?

Thanks,

Komal

[Komal Thareja]

Hello Nagmat,

I suspect your bastion keys are expired and hence the failure.

Could you please run the notebook: jupyter-examples-rel1.6.1/configure_and_validate.ipynb ?

This should update your bastion keys if expired. Please try SSH access again after that from console as well as the notebook. Please restart the notebook kernel after the keys are updated.

Thanks,

Komal

[Komal Thareja]

Just realized you also had problem with network reachability between Chameleon and Fabric nodes. Could you please share your slice ID for FABRIC?

Also, please check the interface and routes are setup correctly on Chameleon node.

[Komal Thareja]

@Sanjana – Chameleon team would be better equipped to help you regarding the failure observed while creating Node on Chameleon. Jupyter Notebook referred in your post uses chameleon python API.

FABRIC portal doesn’t provide support to provision resources on Chameleon. You would have to use Chameleon Portal to use their Graphical Interface.

Thanks,

Komal

[Komal Thareja]

Maintenance is completed and testbed is ready to use. Happy Slicing on FABRIC!

[Author]

Posts