[Komal Thareja]

Hello Nirmala,

Over the weekend, we encountered memory failures on the Wash workers, necessitating their reboot. Unfortunately, this led to the loss of IP addresses of your VMs. Rest assured, we are actively addressing the memory failure issue to prevent further worker reboots.

In the meantime, you can utilize the following block in a notebook to restore your IP configuration without having to delete your slice. We apologize for any inconvenience this may have caused.

try:
slice = fablib.get_slice(name=slice_name)
for node in slice.get_nodes():
print(f"{node}")
node.config()
except Exception as e:
print(f"Exception: {e}")

Thank you for your understanding,

Komal

• This reply was modified 1 year, 2 months ago by Komal Thareja.

[Komal Thareja]

@Vaiden, @Nirmala,

The issue has been resolved. Jupyter Hub is accessible now. Please let us know if you still run into any issues.

Thanks,

Komal

[Komal Thareja]

This issue has been resolved and Jupyter Hub is accessible again.

Thanks,

Komal

[Komal Thareja]

Hi Nirmala,

Thank you for reporting this. It looks like ours K8s cluster hosting Jupyter Hub is down. We are working to resolve this and will keep you posted.

Thanks,

Komal

[Komal Thareja]

Hi Jacob,

I used nslookup to determine the FQDN for your server and can confirm that I can ping your host as shown below.
SALT is IPv6-only site. I will check and confirm if FABRIC NAT server config needs changes to enable this. But the reachability is working with FQDN/hostname.


root@TransferNode:~# nslookup 129.114.108.207
207.108.114.129.in-addr.arpa name = chi-dyn-129-114-108-207.tacc.chameleoncloud.org.


root@TransferNode:~#
root@TransferNode:~#
root@TransferNode:~#
root@TransferNode:~# ping chi-dyn-129-114-108-207.tacc.chameleoncloud.org
PING chi-dyn-129-114-108-207.tacc.chameleoncloud.org(chi-dyn-129-114-108-207.tacc.chameleoncloud.org (2600:2701:5000:5001::8172:6ccf)) 56 data bytes
64 bytes from chi-dyn-129-114-108-207.tacc.chameleoncloud.org (2600:2701:5000:5001::8172:6ccf): icmp_seq=1 ttl=35 time=113 ms
64 bytes from chi-dyn-129-114-108-207.tacc.chameleoncloud.org (2600:2701:5000:5001::8172:6ccf): icmp_seq=2 ttl=35 time=113 ms
64 bytes from chi-dyn-129-114-108-207.tacc.chameleoncloud.org (2600:2701:5000:5001::8172:6ccf): icmp_seq=3 ttl=35 time=113 ms
64 bytes from chi-dyn-129-114-108-207.tacc.chameleoncloud.org (2600:2701:5000:5001::8172:6ccf): icmp_seq=4 ttl=35 time=113 ms
64 bytes from chi-dyn-129-114-108-207.tacc.chameleoncloud.org (2600:2701:5000:5001::8172:6ccf): icmp_seq=5 ttl=35 time=113 ms
64 bytes from chi-dyn-129-114-108-207.tacc.chameleoncloud.org (2600:2701:5000:5001::8172:6ccf): icmp_seq=6 ttl=35 time=113 ms
64 bytes from chi-dyn-129-114-108-207.tacc.chameleoncloud.org (2600:2701:5000:5001::8172:6ccf): icmp_seq=7 ttl=35 time=113 ms
64 bytes from chi-dyn-129-114-108-207.tacc.chameleoncloud.org (2600:2701:5000:5001::8172:6ccf): icmp_seq=8 ttl=35 time=113 ms
64 bytes from chi-dyn-129-114-108-207.tacc.chameleoncloud.org (2600:2701:5000:5001::8172:6ccf): icmp_seq=9 ttl=35 time=113 ms
64 bytes from chi-dyn-129-114-108-207.tacc.chameleoncloud.org (2600:2701:5000:5001::8172:6ccf): icmp_seq=10 ttl=35 time=113 ms


Thanks,
Komal

[Komal Thareja]

Hi Jacob,

I noticed that /etc/resolv.confwas updated on your VM probably via nat64.sh. I reverted it back to the default as shown below. Your original file is saved as /etc/resolv.conf.bkp.

With this change, I was able to ping github.com an IPV4 domain. IPv4 subnets should be reachable. Please note nat64.sh is no longer required. I will update the Knowledge base article also to reflect this.

root@TransferNode:/etc# cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search openstacklocal

root@TransferNode:/etc# ping -c2 github.com
PING github.com(lb-140-82-112-4-iad.github.com (2600:2701:5000:5001::8c52:7004)) 56 data bytes
64 bytes from lb-140-82-112-4-iad.github.com (2600:2701:5000:5001::8c52:7004): icmp_seq=1 ttl=230 time=88.4 ms
64 bytes from lb-140-82-112-4-iad.github.com (2600:2701:5000:5001::8c52:7004): icmp_seq=2 ttl=230 time=87.4 ms

--- github.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 87.439/87.930/88.422/0.491 ms
root@TransferNode:/etc#

Thanks,

Komal

• This reply was modified 1 year, 2 months ago by Komal Thareja.

[Komal Thareja]

Hi Jacob,

Thank you for clarifying your setup. FABRIC is now running it’s own NAT gateway. All VMs are configured to use it for NAT resolution so IPv4 addresses should be accessible.

Could you please share your slice ID? I’ll check your slice and share my findings.

Thanks,

Komal

• This reply was modified 1 year, 2 months ago by Komal Thareja.

[Komal Thareja]

FABRIC only allows SSH and few ICMP messages over the management interface. Hosting services on management network is not recommended. Instead, we recommend using data plane network for your service.

FABRIC serves as a secure sandbox, allowing students and researchers to experiment with potentially disruptive and vulnerable software architectures in a protected environment. When connecting external devices, such as laptops or servers, to nodes within a slice, it is crucial to employ secure methods like SSH tunnels. A Jupyter notebook example illustrates how to create SSH tunnels through the FABRIC bastion host. Alternatively, users can utilize personal VPNs like Tailscale for secure connections.

Exposing ports to the entire Internet is restricted, reserved only for exceptional cases where alternative solutions are not viable. Moreover, users undertaking such capabilities are responsible for deploying, maintaining, and ensuring the security of experiments, akin to a production data center. IPv4Ext and IPv6Ext services facilitate these capabilities.

For newcomers, getting acquainted with SSH tunnels is recommended due to their simplicity and security. If users have additional questions or require further guidance, they are encouraged to reach out.

Thanks,

Komal

[Komal Thareja]

Maintenance is complete and network model has been updated.

Thanks,

Komal

[Komal Thareja]

Hello,

Just to add to what Luis said, it is possible to view slices of other project members via JH using the notebook: jupyter-examples-main/fabric_examples/fablib_api/slice_sharing/slice_sharing.ipynb

The above note book is listed as Share Slices notebook on start_here.ipynb

In addition, an user’s SSH keys can be added to a Slice VMs to give them access via: jupyter-examples-main/fabric_examples/fablib_api/ssh_keys/ssh_keys.ipynb

For now, only the slice owner can extend/renew their slice.

Thanks,

Komal

• This reply was modified 1 year, 2 months ago by Komal Thareja.
• This reply was modified 1 year, 2 months ago by Komal Thareja.

1 user thanked author for this post.

Garegin Grigoryan

[Komal Thareja]

This link between INDI and STAR has been restored. FabNet services on INDI are working now.

Thanks,

Komal

[Komal Thareja]

Hi Manas,

Glad to hear the issue is resolved. Please find my responses inline below.

1. I have recently changed the bastion key, so I don’t get why the key expired.

[KT] Could you please share how this was done? Please note if the bastion keys are re-created via portal, user is responsible to upload them to JH.

1. I am able to ssh into the node, but the bastion key I have in my laptop (local) is different from the bastion key in the Jupyter hub.

[KT] This should not be an issue. Users can have multiple bastion keys.

1. Is multiple bastion keys possible?

[KT] Yes, this not an issue.

1. How to check the expiration date of the bastion key in my Jupyter hub, I know how to check the expiration date of the bastion key that I use to ssh into the node from my local machine.

[KT] You can check the bastion key expiry on the portal or run the validate_and_configure.ipynb to validate/update your config as needed.

Hope this helps!

Thanks,

Komal

[Komal Thareja]

Hi Manas,

This typically happens when either your bastion keys are expired or sliver keys used by fablib don’t match the keys inside the slivers.

Could you please run this notebook: jupyter-examples-rel1.6.1/configure_and_validate.ipynb to ensure your config is valid and if bastion keys are expired they are regenerated?

Please try list/show on your slice after that and let us know if you still face the problem.

P.S: Verified that all your VMs are accessible via SSH.

Thanks,

Komal

• This reply was modified 1 year, 3 months ago by Komal Thareja.

[Komal Thareja]

Hello Zheyi,

The VMs from your slices are being provisioned on eduky-w12.fabric-testbed.net This host is heavily used and is reporting Page Faults and Memory issues. We are investigating that and have placed eduky-w12.fabric-testbed.net in Maintenance.

Please try creating a slice on EDUKY and let us know if you still see the issue.

Thanks,

Komal

1 user thanked author for this post.

Zheyi Qin

[Komal Thareja]

Good morning Violet and Jackson,

We have deployed a fix for the OVS Bridges. Experiments with OVS Bridges can now be conducted using NIC_Basic with the following host considerations.

 

Host Considerations:

Because of constraints imposed by NVIDIA/Mellanox, when utilizing NIC_Basic for an OVS bridge experiment, it is advisable to deploy the VM responsible for running the bridge on a separate host from the VMs linked to the bridge.

Additionally, it’s worth noting that this condition does not apply to NIC_ConnectX_5 and NIC_ConnectX_6 configurations.

 

Example Notebook:

Updated example is available at: https://github.com/fabric-testbed/jupyter-examples/blob/main/fabric_examples/complex_recipes/openvswitch/openvswitch.ipynb

Thanks,

Komal

[Author]

Posts