Active VM at SALT unreachable through management network

[DANILO RENATO DE ASSIS]

Hello FABRIC Support,

I have an active node containing important research data that has become unreachable through the FABRIC management network.

Slice: NS1
Node: NS1
Node reservation ID: 4d8d7d4a-f188-4ecc-9e6e-741c85cb7d3f
Site: SALT
Host: salt-w1.fabric-testbed.net
Management IP: 2001:400:a100:3010:f816:3eff:fe90:e98
Image: default_ubuntu_22
Disk: 1000 GB
Slice state: StableOK
Node state: Active
FABRIC error message: empty

SSH through the FABRIC bastion consistently fails with:

channel 0: open failed: connect failed: No route to host

fablib node.test_ssh() also returns False.

A POA addkey operation fails with:

Playbook has failed tasks: All items completed

I also performed node.os_reboot(). The reboot request completed, but management connectivity did not recover.

My other FABRIC nodes at WASH, EDC, FIU, GATECH, PSC, AMST and CLEM are all reachable through the same bastion/key configuration, including several nodes using IPv6 management addresses.

There is important research data stored on the NS1 root disk. Please do not delete or reprovision this VM.

Could you please check the management network/port for this VM on salt-w1, or advise whether the VM disk can be preserved or accessed for data recovery if the management network cannot be restored?

The current lease expires on 2026-09-18 22:54:42 UTC.

Thank you

[Mert Cevik]

There was a power outage in the datacenter that caused all servers to reboot. Now, they are online. Their PCI devices are reattached (may require a reboot of the VM if they don’t show up).

[Author]

Posts