Error logging into Nodes

Hi,

Could you check in the portal if your current keys are expired (both slice and bastion keys) ? If keys already exist, I believe the configure environment notebook won’t generate new keys. If they are expired, try deleting the keys and then re-running the configuration notebook again.

Thanks!

Update: I have tried this in both JupyterHub and my laptop without success. I am attaching the verbose output from my ssh command (which I copied from the slice manager) in case it is useful in identifying what is going wrong. From my end, it appears as though the bastion host is simply refusing to accept my public key:

debug1: Will attempt key: <path_to_private_bastion_key> ECDSA SHA256:gjlV8496hVF5KnDaQbr5/JMADpQA30HqZ6va8arcaIw explicit

debug1: Offering public key: <path_to_private_bastion_key> ECDSA SHA256:gjlV8496hVF5KnDaQbr5/JMADpQA30HqZ6va8arcaIw explicit

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic

debug1: No more authentication methods to try.

<username>@bastion.fabric-testbed.net: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

kex_exchange_identification: Connection closed by remote host

Connection closed by UNKNOWN port 65535

Also please note that I tried this with locally generated keys as well as key pairs generated through FABRIC. Neither approach was successful.

Thanks for the response. I tried the Jupyter notebooks you suggested and experienced exactly the same error – the authentication to the bastion host fails when trying to SSH to the node (this is the output from cell 6 in the Hello Fabric notebook):

---------------------------------------------------------------------------
AuthenticationException                   Traceback (most recent call last)
Cell In[6], line 4
      1 #node = slice.get_node('Node1')
      3 for node in slice.get_nodes():
----> 4     stdout, stderr = node.execute('echo Hello, FABRIC from node hostname -s')

File /opt/conda/lib/python3.11/site-packages/fabrictestbed_extensions/fablib/node.py:1520, in Node.execute(self, command, retry, retry_interval, username, private_key_file, private_key_passphrase, quiet, read_timeout, timeout, output_file, display)
   1518 bastion = paramiko.SSHClient()
   1519 bastion.set_missing_host_key_policy(paramiko.AutoAddPolicy())
-> 1520 bastion.connect(
   1521     fablib_manager.get_bastion_host(),
   1522     username=bastion_username,
   1523     key_filename=bastion_key_file,
   1524 )
   1526 bastion_transport = bastion.get_transport()
   1527 bastion_channel = bastion_transport.open_channel(
   1528     "direct-tcpip", dest_addr, src_addr
   1529 )

File /opt/conda/lib/python3.11/site-packages/paramiko/client.py:485, in SSHClient.connect(self, hostname, port, username, password, pkey, key_filename, timeout, allow_agent, look_for_keys, compress, sock, gss_auth, gss_kex, gss_deleg_creds, gss_host, banner_timeout, auth_timeout, channel_timeout, gss_trust_dns, passphrase, disabled_algorithms, transport_factory, auth_strategy)
    482 else:
    483     key_filenames = key_filename
--> 485 self._auth(
    486     username,
    487     password,
    488     pkey,
    489     key_filenames,
    490     allow_agent,
    491     look_for_keys,
    492     gss_auth,
    493     gss_kex,
    494     gss_deleg_creds,
    495     t.gss_host,
    496     passphrase,
    497 )

File /opt/conda/lib/python3.11/site-packages/paramiko/client.py:818, in SSHClient._auth(self, username, password, pkey, key_filenames, allow_agent, look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host, passphrase)
    816 # if we got an auth-failed exception earlier, re-raise it
    817 if saved_exception is not None:
--> 818     raise saved_exception
    819 raise SSHException("No authentication methods available")

File /opt/conda/lib/python3.11/site-packages/paramiko/client.py:794, in SSHClient._auth(self, username, password, pkey, key_filenames, allow_agent, look_for_keys, gss_auth, gss_kex, gss_deleg_creds, gss_host, passphrase)
    788 key = self._key_from_filepath(
    789     filename, pkey_class, passphrase
    790 )
    791 # for 2-factor auth a successfully auth'd key will result
    792 # in ['password']
    793 allowed_types = set(
--> 794     self._transport.auth_publickey(username, key)
    795 )
    796 two_factor = allowed_types & two_factor_types
    797 if not two_factor:

File /opt/conda/lib/python3.11/site-packages/paramiko/transport.py:1709, in Transport.auth_publickey(self, username, key, event)
   1706 if event is not None:
   1707     # caller wants to wait for event themselves
   1708     return []
-> 1709 return self.auth_handler.wait_for_response(my_event)

File /opt/conda/lib/python3.11/site-packages/paramiko/auth_handler.py:263, in AuthHandler.wait_for_response(self, event)
    261     if issubclass(e.__class__, PartialAuthentication):
    262         return e.allowed_types
--> 263     raise e
    264 return []

AuthenticationException: Authentication failed.