Subject: Issues with SSH Access to Fabric Nodes for Slice IDs: 34c41dad-c9f3-431

[Yuanjun Dai]

Dear Fabric Support Team,

I am experiencing issues with SSH access to my Fabric nodes associated with the following slice IDs:

• 34c41dad-c9f3-431b-90e1-37a77491130b
• d021c6c7-e8ca-48ad-ba24-120f0bae2e97
• 9e1f043c-87a7-4e7e-a029-3530644e612e
• f5ee47cc-4a2b-434a-8b90-c2850c8fa795

These nodes have been running for over a month without any issues, and I was able to SSH into them as recently as this morning. However, this afternoon, I suddenly lost the ability to log in. All nodes are now returning the following error:

Warning: Permanently added ‘bastion.fabric-testbed.net,2600:2701:5000:a902::c’ (ECDSA) to the list of known hosts.
yuanjun_dai_0033333176@bastion.fabric-testbed.net: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
kex_exchange_identification: Connection closed by remote host

It seems like there might be an issue with my SSH key, although I haven’t made any changes to it.

Here’s what I’ve checked so far:

1. SSH Key: My SSH key was working perfectly until today, and I haven’t modified it recently.
2. Slice Status: These slices have been active for over a month, and I am concerned there might be an issue with their current status or a possible automatic configuration change on the Fabric platform.
3. Platform and Network: I have not encountered any known network changes on my end that could have affected connectivity.

Could you please assist in diagnosing this issue? I need to regain access to these nodes as soon as possible.

Thank you for your support.

Best regards,

Yuanjun Dai

[Komal Thareja]

Hi Yuanjun,

I suspect your bastion keys have expired. Could you please re-run the notebook to regenerate your bastion keys jupyter-examples-rel1.7.0/configure_and_validate.ipynb ?

Please let us know if you still run into errors.

Thanks,

Komal

[Yuanjun Dai]

Hi, does it mean, I need to create and update a new key?

However, the key in the host is still old. Do I need to create new slices?

[Yuanjun Dai]

User: yuanjun.dai@case.edu bastion key is valid!
Configuration is valid and please save the config!
Exception ignored in: <function Pool.__del__ at 0x7f133dbef2e0>
Traceback (most recent call last):
File “/home/johnnydai/miniconda3/lib/python3.11/multiprocessing/pool.py”, line 271, in __del__
File “/home/johnnydai/miniconda3/lib/python3.11/multiprocessing/queues.py”, line 371, in put
AttributeError: ‘NoneType’ object has no attribute ‘dumps’
Exception ignored in: <function Pool.__del__ at 0x7f133dbef2e0>
Traceback (most recent call last):
File “/home/johnnydai/miniconda3/lib/python3.11/multiprocessing/pool.py”, line 271, in __del__
File “/home/johnnydai/miniconda3/lib/python3.11/multiprocessing/queues.py”, line 371, in put
AttributeError: ‘NoneType’ object has no attribute ‘dumps’

[Komal Thareja]

Hi Yuanjun,

I suspect the bastion keys are expired. These keys are only on your JH container and are not pushed to your VMs. Sliver keys i.e. VM keys should not be affected. The error indicated above for SSH failure indicates login to bastion server was denied and hence the suggestion.

 

The error observed in the multi-processing pool cleanup can be ignored. We will address that error but it should not impact regeneration of the keys. Could you please see if you are able to SSH to your VMs?

Bastion key expiry can also be verified from the portal via: Experiments -> Manage SSH Keys.

Thanks,

Komal

[Author]

Posts