SSH error: channel 0: open failed: connect failed: No route to host

Hello everyone,

Lately, I have been seeing this error more often.

When I SSH directly using:

ssh -i /home/ubuntu/work/fabric_config/slice_key -F /home/ubuntu/work/fabric_config/ssh_config ubuntu@2001:400:a100:3010:f816:3eff:fe19:dd2a

I get:

Warning: Permanently added 'bastion.fabric-testbed.net' (ED25519) to the list of known hosts.
channel 0: open failed: connect failed: No route to host
stdio forwarding failed
Connection closed by UNKNOWN port 65535

I also see a similar issue when using node.execute():

ERROR:paramiko.transport:Secsh channel 0 open FAILED: No route to host: Connect failed
Server21: collection error: ChannelException(2, 'Connect failed')

Sometimes the entire slice has this issue, while other times only some nodes (VMs) are affected. In many cases, the problem lasts for about 12 hours and then goes away. After that, I often need to reconfigure some parameters on those VMs. I suspect the node may have rebooted, which makes it reachable again.

Is there a way to avoid this issue? Also, are there any immediate commands or actions I can use to recover broken nodes right away instead of waiting around 12 hours?

Currently, I am seeing this issue on all VMs in the following slice:

sliceID = e9df9e9c-637b-4498-854b-6a192326c8c6

Thank you,